idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-bidirection-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 27, 2016) is 2862 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-04 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Standards Track May 27, 2016 5 Expires: November 28, 2016 7 Bi-directional Remote Procedure Call On RPC-over-RDMA Transports 8 draft-ietf-nfsv4-rpcrdma-bidirection-04 10 Abstract 12 Minor versions of NFSv4 newer than NFSv4.0 work best when ONC RPC 13 transports can send Remote Procedure Call transactions in both 14 directions on the same connection. This document describes how RPC- 15 over-RDMA transport endpoints convey RPCs in both directions on a 16 single connection. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on November 28, 2016. 35 Copyright Notice 37 Copyright (c) 2016 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 2. Understanding RPC Direction . . . . . . . . . . . . . . . . . 3 55 2.1. Forward Direction . . . . . . . . . . . . . . . . . . . . 3 56 2.2. Backward Direction . . . . . . . . . . . . . . . . . . . 4 57 2.3. Bi-directional Operation . . . . . . . . . . . . . . . . 4 58 2.4. XID Values . . . . . . . . . . . . . . . . . . . . . . . 4 59 3. Immediate Uses Of Bi-Directional RPC-over-RDMA . . . . . . . 5 60 3.1. NFSv4.0 Callback Operation . . . . . . . . . . . . . . . 5 61 3.2. NFSv4.1 Callback Operation . . . . . . . . . . . . . . . 6 62 4. Flow Control . . . . . . . . . . . . . . . . . . . . . . . . 6 63 4.1. Backward Credits . . . . . . . . . . . . . . . . . . . . 7 64 4.2. Inline Thresholds . . . . . . . . . . . . . . . . . . . . 7 65 4.3. Managing Receive Buffers . . . . . . . . . . . . . . . . 7 66 5. Sending And Receiving Backward Operations . . . . . . . . . . 8 67 5.1. Sending A Backward Direction Call . . . . . . . . . . . . 9 68 5.2. Sending A Backward Direction Reply . . . . . . . . . . . 9 69 5.3. Backward Direction Chunks . . . . . . . . . . . . . . . . 9 70 5.4. Backward Direction Retransmission . . . . . . . . . . . . 10 71 6. In the Absence of Backward Direction Support . . . . . . . . 11 72 7. Considerations For Upper Layer Bindings . . . . . . . . . . . 11 73 8. Security Considerations . . . . . . . . . . . . . . . . . . . 12 74 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 75 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 76 11. Normative References . . . . . . . . . . . . . . . . . . . . 12 77 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 79 1. Introduction 81 The purpose of this document is to enable concurrent operation in 82 both directions on a single transport connection using RPC-over-RDMA 83 protocol versions that do not have specific facilities for backward 84 direction operation. 86 Backward direction RPC transactions are necessary for the operation 87 of NFSv4.1, and in particular, of pNFS, though any Upper Layer 88 Protocol implementation may make use of them. An Upper Layer Binding 89 for NFSv4.x callback operation is additionally required (see 90 Section 7), but is not provided in this document. 92 For example, using the approach described herein, RPC transactions 93 can be conveyed in both directions on the same RPC-over-RDMA Version 94 One connection without changes to the the XDR description of RPC- 95 over-RDMA Version One. This document does not modify the XDR or 96 protocol described in [I-D.ietf-nfsv4-rfc5666bis]. Future versions 97 of RPC-over-RDMA may adopt the approach described herein, or may 98 replace it with a different approach. 100 1.1. Requirements Language 102 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 103 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 104 document are to be interpreted as described in [RFC2119]. 106 2. Understanding RPC Direction 108 The ONC RPC protocol as described in [RFC5531] is architected as a 109 message-passing protocol between one server and one or more clients. 110 ONC RPC transactions are made up of two types of messages. 112 A CALL message, or "Call", requests work. A Call is designated by 113 the value CALL in the message's msg_type field. An arbitrary unique 114 value is placed in the message's xid field. A host that originates a 115 Call is referred to in this document as a "Requester." 117 A REPLY message, or "Reply", reports the results of work requested by 118 a Call. A Reply is designated by the value REPLY in the message's 119 msg_type field. The value contained in the message's xid field is 120 copied from the Call whose results are being returned. A host that 121 emits a Reply is referred to as a "Responder." 123 Typically, a Call results in a corresponding Reply. A Reply is never 124 sent without a corresponding Call. 126 RPC-over-RDMA is a connection-oriented RPC transport. In all cases, 127 when a connection-oriented transport is used, ONC RPC client 128 endpoints are responsible for initiating transport connections, while 129 ONC RPC service endpoints passively await incoming connection 130 requests. 132 RPC direction on connectionless RPC transports is not addressed in 133 this document. 135 2.1. Forward Direction 137 Traditionally, an ONC RPC client acts as a Requester, while an ONC 138 RPC service acts as a Responder. This form of message passing is 139 referred to as "forward direction" operation. 141 2.2. Backward Direction 143 The ONC RPC specification [RFC5531] does not forbid passing messages 144 in the other direction. An ONC RPC service endpoint can act as a 145 Requester, in which case an ONC RPC client endpoint acts as a 146 Responder. This form of message passing is referred to as "backward 147 direction" operation. 149 During backward direction operation, the ONC RPC client is 150 responsible for establishing transport connections, even though ONC 151 RPC Calls come from the ONC RPC server. 153 ONC RPC clients and services are optimized to perform and scale well 154 while handling traffic in the forward direction, and may not be 155 prepared to handle operation in the backward direction. Not until 156 NFSv4.1 [RFC5661] has there been a strong need to handle backward 157 direction operation. 159 2.3. Bi-directional Operation 161 A pair of connected RPC endpoints may choose to use only forward or 162 only backward direction operations on a particular transport. Or, 163 these endpoints may send Calls in both directions concurrently on the 164 same transport. 166 "Bi-directional operation" occurs when both transport endpoints act 167 as a Requester and a Responder at the same time. 169 Bi-directionality is an extension of RPC transport connection 170 sharing. Two RPC endpoints wish to exchange independent RPC messages 171 over a shared connection, but in opposite directions. These messages 172 may or may not be related to the same workloads or RPC Programs. 174 2.4. XID Values 176 Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, 177 or "xid" for short. The value of an xid is interpreted in the 178 context of the message's msg_type field. 180 o The xid of a Call is arbitrary but is unique among outstanding 181 Calls from that Requester. 183 o The xid of a Reply always matches that of the initiating Call. 185 When receiving a Reply, a Requester matches the xid value in the 186 Reply with a Call it previously sent. 188 2.4.1. XID Generation 190 During bi-directional operation, forward and backward direction XIDs 191 are typically generated on distinct hosts by possibly different 192 algorithms. There is no co-ordination between forward and backward 193 direction XID generation. 195 Therefore, a forward direction Requester MAY use the same xid value 196 at the same time as a backward direction Requester on the same 197 transport connection. Though such concurrent requests use the same 198 xid value, they represent distinct ONC RPC transactions. 200 3. Immediate Uses Of Bi-Directional RPC-over-RDMA 202 3.1. NFSv4.0 Callback Operation 204 An NFSv4.0 client employs a traditional ONC RPC client to send NFS 205 requests to an NFSv4.0 server's traditional ONC RPC service 206 [RFC7530]. NFSv4.0 requests flow in the forward direction on a 207 connection established by the client. This connection is referred to 208 as a "forechannel" connection. 210 An NFSv4 "delegation" is simply a promise made by a server that it 211 will notify a client before another client or program running on the 212 server is allowed access to a file. With this guarantee, that client 213 can operate as sole accessor of the file. In particular, it can 214 manage the file's data and metadata caches aggressively. 216 To administer file delegations, NFSv4.0 introduces the use of 217 callback operations, or "callbacks", in Section 10.2 of [RFC7530]. 218 An NFSv4.0 server sets up a traditional ONC RPC client, and an 219 NFSv4.0 client sets up a traditional ONC RPC service. Callbacks flow 220 in the forward direction on a connection established between the 221 server's callback client, and the client's callback server. This 222 connection is distinct from connections being used as forechannels, 223 and is referred to as a "backchannel connection." 225 When an RDMA transport is used as a forechannel, an NFSv4.0 client 226 typically provides a TCP callback service. The client's SETCLIENTID 227 operation advertises the callback service endpoint with a "tcp" or 228 "tcp6" netid. The server then connects to this service using a TCP 229 socket. 231 NFSv4.0 implementations can function without a backchannel in place. 232 In this case, the server does not grant file delegations. This might 233 result in a negative performance effect, but correctness is not 234 affected. 236 3.2. NFSv4.1 Callback Operation 238 NFSv4.1 supports file delegation in a similar fashion to NFSv4.0, and 239 extends the callback mechanism to manage pNFS layouts, as discussed 240 in Section 12 of [RFC5661]. 242 To facilitate operation through NAT routers, all NFSv4.1 transport 243 connections are initiated by NFSv4.1 clients. Therefore NFSv4.1 244 servers send callbacks to clients in the backward direction on 245 connections established by NFSv4.1 clients. 247 NFSv4.1 clients and servers indicate to their peers that a 248 backchannel capability is available on a given transport in the 249 arguments and results of NFS CREATE_SESSION or BIND_CONN_TO_SESSION 250 operations. 252 NFSv4.1 clients may establish distinct transport connections for 253 forechannel and backchannel operation, or they may combine 254 forechannel and backchannel operation on one transport connection 255 using bi-directional operation. 257 Without a backward direction RPC-over-RDMA capability, an NFSv4.1 258 client must additionally connect using a transport with backward 259 direction capability to use as a backchannel. TCP is the only choice 260 for an NFSv4.1 backchannel connection in this case. 262 Implementations often find it more convenient to use a single 263 combined transport (ie. a transport that is capable of bi-directional 264 operation). This simplifies connection establishment and recovery 265 during network partitions or when one endpoint restarts. This can 266 also enable better scaling by using fewer transport connections to 267 perform the same work. 269 As with NFSv4.0, if a backchannel is not in use, an NFSv4.1 server 270 does not grant delegations. Because NFSv4.1 relies on callbacks to 271 manage pNFS layout state, pNFS operation is not possible without a 272 backchannel. 274 4. Flow Control 276 For an RDMA Send operation to work properly, the receiving peer must 277 have posted a receive buffer in which to accept the incoming message. 278 If a receiver hasn't posted enough buffers to accommodate each 279 incoming Send operation, the receiving RDMA provider is allowed to 280 terminate the RDMA connection. 282 RPC-over-RDMA transport protocols provide built-in send flow control 283 to prevent overrunning the number of pre-posted receive buffers on a 284 connection's receive endpoint. For RPC-over-RDMA Version One, this 285 is discussed in Section 4.3 of [I-D.ietf-nfsv4-rfc5666bis]. 287 4.1. Backward Credits 289 Credits work the same way in the backward direction as they do in the 290 forward direction. However, forward direction credits and backward 291 direction credits are accounted separately. 293 In other words, the forward direction credit value is the same 294 whether or not there are backward direction resources associated with 295 an RPC-over-RDMA transport connection. The backward direction credit 296 value MAY be different than the forward direction credit value. The 297 rdma_credit field in a backward direction RPC-over-RDMA message MUST 298 NOT contain the value zero. 300 A backward direction Requester (ie, an RPC-over-RDMA service 301 endpoint) requests credits from the Responder (ie, an RPC-over-RDMA 302 client endpoint). The Responder reports how many credits it has 303 granted. This is the number of backward direction Calls the 304 Responder is prepared to handle at once. 306 When message direction is not fully determined by context or by an 307 accompanying RPC message with a call direction field, it is not 308 possible to tell whether the header credit value is a request or 309 grant, or whether the value applies to the forward direction or 310 backward direction. In such cases, the receiver MUST NOT use the 311 header's credit value. 313 4.2. Inline Thresholds 315 Forward and backward operation on the same connection share the same 316 receive buffers. Therefore the inline threshold values for the 317 forward direction and the backward direction are the same. The call 318 inline threshold for the backward direction is the same as the reply 319 inline threshold for the forward direction, and vice versa. For more 320 information, see Section 4.3.2 of [I-D.ietf-nfsv4-rfc5666bis]. 322 4.3. Managing Receive Buffers 324 An RPC-over-RDMA transport endpoint must pre-post receive buffers 325 before it can receive and process incoming RPC-over-RDMA messages. 326 If a sender transmits a message for a receiver which has no posted 327 receive buffer, the RDMA provider is allowed to drop the RDMA 328 connection. 330 4.3.1. Client Receive Buffers 332 Typically an RPC-over-RDMA Requester posts only as many receive 333 buffers as there are outstanding RPC Calls. A client endpoint 334 without backward direction support might therefore at times have no 335 pre-posted receive buffers. 337 To receive incoming backward direction Calls, an RPC-over-RDMA client 338 endpoint must pre-post enough additional receive buffers to match its 339 advertised backward direction credit value. Each outstanding forward 340 direction RPC requires an additional receive buffer above this 341 minimum. 343 When an RDMA transport connection is lost, all active receive buffers 344 are flushed and are no longer available to receive incoming messages. 345 When a fresh transport connection is established, a client endpoint 346 must re-post a receive buffer to handle the Reply for each 347 retransmitted forward direction Call, and a full set of receive 348 buffers to handle backward direction Calls. 350 4.3.2. Server Receive Buffers 352 A forward direction RPC-over-RDMA service endpoint posts as many 353 receive buffers as it expects incoming forward direction Calls. That 354 is, it posts no fewer buffers than the number of credits granted in 355 the rdma_credit field of forward direction RPC replies. 357 To receive incoming backward direction replies, an RPC-over-RDMA 358 server endpoint must pre-post a receive buffer for each backward 359 direction Call it sends. 361 When the existing transport connection is lost, all active receive 362 buffers are flushed and are no longer available to receive incoming 363 messages. When a fresh transport connection is established, a server 364 endpoint must re-post a receive buffer to handle the Reply for each 365 retransmitted backward direction Call, and a full set of receive 366 buffers for receiving forward direction Calls. 368 5. Sending And Receiving Backward Operations 370 The operation of RPC-over-RDMA transports in the forward direction is 371 defined in [RFC5531] and [I-D.ietf-nfsv4-rfc5666bis]. In this 372 section, a mechanism for backward direction operation on RPC-over- 373 RDMA is defined. Backward operation used in combination with forward 374 operation enables bi-directional communication on a common RPC 375 transport connection. 377 Certain fields in the RPC-over-RDMA header are fixed for all versions 378 of RPC-over-RDMA. The XDR description of these fields is contained 379 in Section 5.1 of [I-D.ietf-nfsv4-rfc5666bis]. 381 5.1. Sending A Backward Direction Call 383 To form a backward direction RPC-over-RDMA Call message, an ONC RPC 384 service endpoint constructs an RPC-over-RDMA header containing a 385 fresh RPC XID in the rdma_xid field (see Section 2.4 for full 386 requirements). 388 The rdma_vers field MUST contain the same value in backward and 389 forward direction Call messages on the same connection. 391 The number of requested backward direction credits is placed in the 392 rdma_credit field (see Section 4). 394 Whether presented inline or as a separate chunk, the ONC RPC Call 395 header MUST start with the same XID value that is present in the RPC- 396 over-RDMA header, and the RPC header's msg_type field MUST contain 397 the value CALL. 399 5.2. Sending A Backward Direction Reply 401 To form a backward direction RPC-over-RDMA Reply message, an ONC RPC 402 client endpoint constructs an RPC-over-RDMA header containing a copy 403 of the matching ONC RPC Call's RPC XID in the rdma_xid field (see 404 Section 2.4 for full requirements). 406 The rdma_vers field MUST contain the same value in a backward 407 direction Reply message as in the matching Call message. 409 The number of granted backward direction credits is placed in the 410 rdma_credit field (see Section 4). 412 Whether presented inline or as a separate chunk, the ONC RPC Reply 413 header MUST start with the same XID value that is present in the RPC- 414 over-RDMA header, and the RPC header's msg_type field MUST contain 415 the value REPLY. 417 5.3. Backward Direction Chunks 419 Chunks MAY be used in the backward direction. They operate the same 420 way as in the forward direction (see [I-D.ietf-nfsv4-rfc5666bis] for 421 details). 423 An implementation might not support any Upper Layer Protocol that has 424 DDP-eligible data items. The Upper Layer Protocol may also use only 425 small messages, or it may have a native mechanism for restricting the 426 size of backward direction RPC messages, obviating the need to handle 427 Long Messages in the backward direction. 429 When there is no Upper Layer Protocol requirement for chunks, 430 implementers can choose not to provide support for chunks in the 431 backward direction. This avoids the complexity of adding support for 432 performing RDMA Reads and Writes in the backward direction. 434 When chunks are not implemented, RPC messages in the backward 435 direction are always sent using RDMA_MSG, and therefore can be no 436 larger than what can be sent inline (that is, without chunks). 437 Sending an inline message larger than the inline threshold can result 438 in loss of connection. 440 If a backward direction requester provides a non-empty chunk list to 441 a responder that does not support chunks, the responder MUST reply 442 with an RDMA_ERROR message with rdma_err field set to ERR_CHUNK. 444 5.4. Backward Direction Retransmission 446 In rare cases, an ONC RPC transaction cannot be completed within a 447 certain time. This can be because the transport connection was lost, 448 the Call or Reply message was dropped, or because the Upper Layer 449 consumer delayed or dropped the ONC RPC request. Typically, the 450 Requester sends the transaction again, reusing the same RPC XID. 451 This is known as an "RPC retransmission". 453 In the forward direction, the Requester is the ONC RPC client. The 454 client is always responsible for establishing a transport connection 455 before sending again. 457 In the backward direction, the Requester is the ONC RPC server. 458 Because an ONC RPC server does not establish transport connections 459 with clients, it cannot send a retransmission if there is no 460 transport connection. It must wait for the ONC RPC client to re- 461 establish the transport connection before it can retransmit ONC RPC 462 transactions in the backward direction. 464 If an ONC RPC client has no work to do, it may be some time before it 465 re-establishes a transport connection. Backward direction Requesters 466 must be prepared to wait indefinitely for a connection to be 467 established before a pending backward direction ONC RPC Call can be 468 retransmitted. 470 Forward direction Requesters are responsible for maintaining a 471 transport connection as long as there is the possibility of backward 472 direction requests. For example, an NFSv4.1 client with open 473 delegated files or active pNFS layouts should maintain a transport 474 connection so the server can send callback operations. 476 6. In the Absence of Backward Direction Support 478 An RPC-over-RDMA transport endpoint might not support backward 479 direction operation (and thus bi-directional operation). There might 480 be no mechanism in the transport implementation to do so. Or in an 481 implementation that can support operation in the backward direction, 482 the Upper Layer Protocol consumer might not yet have configured or 483 enabled the transport to handle backward direction traffic. 485 If an endpoint is not prepared to receive an incoming backward 486 direction message, loss of the RDMA connection might result. Thus 487 denial of service could result if a sender continues to send backward 488 direction messages after every transport reconnect to an endpoint 489 that is not prepared to receive them. 491 When dealing with the possibility that the remote peer has no 492 transport level support for backward direction operation, the Upper 493 Layer Protocol becomes responsible for informing peers when backward 494 direction operation is supported. Otherwise even a simple backward 495 direction NULL probe from a peer could result in a lost connection. 497 Therefore, an Upper Layer Protocol consumer MUST NOT perform backward 498 direction ONC RPC operations unless the peer consumer has indicated 499 it is prepared to handle them. A description of Upper Layer Protocol 500 mechanisms used for this indication is outside the scope of this 501 document. 503 For example, an NFSv4.1 server does not send backchannel messages to 504 an NFSv4.1 client before the NFSv4.1 client has sent a CREATE_SESSION 505 or a BIND_CONN_TO_SESSION operation. As long as an NFSv4.1 client 506 has prepared appropriate backchannel resources before sending one of 507 these operations announcing support for backchannel operation, denial 508 of service is avoided. 510 7. Considerations For Upper Layer Bindings 512 An Upper Layer Protocol that operates on RPC-over-RDMA transports may 513 have procedures that include DDP-eligible data items. DDP- 514 eligibility is specified in an Upper Layer Binding. Direction of 515 operation does not obviate the need for DDP-eligibility statements. 517 Backward-only operation requires the client endpoint to establish a 518 fresh connection. The Upper Layer Binding can specify appropriate 519 RPC binding parameters for such connections. 521 Bi-directional operation occurs on an already-established connection. 522 Specification of RPC binding parameters is usually not necessary in 523 this case. 525 For bi-directional operation, other considerations about sharing an 526 RPC-over-RDMA transport with another ULP may apply. Consult 527 Section 7 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else 528 may be contained in an Upper Layer Binding. 530 8. Security Considerations 532 Security considerations for operation on RPC-over-RDMA transports are 533 outlined in Section 9 of [I-D.ietf-nfsv4-rfc5666bis]. 535 9. IANA Considerations 537 This document does not require actions by IANA. 539 10. Acknowledgements 541 Tom Talpey was an indispensable resource, in addition to creating the 542 foundation upon which this work is based. Our warmest regards go to 543 him for his help and support. 545 Dave Noveck provided excellent review, constructive suggestions, and 546 navigational guidance throughout the process of drafting this 547 document. 549 Dai Ngo was a solid partner and collaborator. Together we 550 constructed and tested independent prototypes of the changes 551 described in this document. 553 The author wishes to thank Bill Baker for his unwavering support of 554 this work. In addition, the author gratefully acknowledges the 555 expert contributions of Karen Deitke, Chunli Zhang, Mahesh 556 Siddheshwar, Steve Wise, and Tom Tucker. 558 Special thanks go to the nfsv4 Working Group Chair Spencer Shepler 559 and the nfsv4 Working Group Secretary Tom Haynes for their support. 561 11. Normative References 563 [I-D.ietf-nfsv4-rfc5666bis] 564 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 565 Memory Access Transport for Remote Procedure Call", draft- 566 ietf-nfsv4-rfc5666bis-04 (work in progress), March 2016. 568 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 569 Requirement Levels", BCP 14, RFC 2119, March 1997. 571 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 572 Specification Version 2", RFC 5531, May 2009. 574 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 575 System (NFS) Version 4 Minor Version 1 Protocol", RFC 576 5661, January 2010. 578 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 579 Version 4 Protocol", RFC 7530, March 2015. 581 Author's Address 583 Charles Lever 584 Oracle Corporation 585 1015 Granger Avenue 586 Ann Arbor, MI 48104 587 USA 589 Phone: +1 734 274 2396 590 Email: chuck.lever@oracle.com