idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-bidirection-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 9, 2016) is 2876 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-04 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Standards Track June 9, 2016 5 Expires: December 11, 2016 7 Bi-directional Remote Procedure Call On RPC-over-RDMA Transports 8 draft-ietf-nfsv4-rpcrdma-bidirection-05 10 Abstract 12 Minor versions of NFSv4 newer than NFSv4.0 work best when ONC RPC 13 transports can send Remote Procedure Call transactions in both 14 directions on the same connection. This document describes how RPC- 15 over-RDMA transport endpoints convey RPCs in both directions on a 16 single connection. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on December 11, 2016. 35 Copyright Notice 37 Copyright (c) 2016 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 2. Understanding RPC Direction . . . . . . . . . . . . . . . . . 3 55 2.1. Forward Direction . . . . . . . . . . . . . . . . . . . . 3 56 2.2. Backward Direction . . . . . . . . . . . . . . . . . . . 4 57 2.3. Bi-directional Operation . . . . . . . . . . . . . . . . 4 58 2.4. XID Values . . . . . . . . . . . . . . . . . . . . . . . 4 59 3. Immediate Uses Of Bi-Directional RPC-over-RDMA . . . . . . . 5 60 3.1. NFSv4.0 Callback Operation . . . . . . . . . . . . . . . 5 61 3.2. NFSv4.1 Callback Operation . . . . . . . . . . . . . . . 6 62 4. Flow Control . . . . . . . . . . . . . . . . . . . . . . . . 6 63 4.1. Backward Credits . . . . . . . . . . . . . . . . . . . . 7 64 4.2. Inline Thresholds . . . . . . . . . . . . . . . . . . . . 7 65 4.3. Managing Receive Buffers . . . . . . . . . . . . . . . . 7 66 5. Sending And Receiving Backward Operations . . . . . . . . . . 8 67 5.1. Sending A Backward Direction Call . . . . . . . . . . . . 9 68 5.2. Sending A Backward Direction Reply . . . . . . . . . . . 9 69 5.3. Backward Direction Chunks . . . . . . . . . . . . . . . . 9 70 5.4. Backward Direction Retransmission . . . . . . . . . . . . 10 71 6. In the Absence of Backward Direction Support . . . . . . . . 11 72 7. Considerations For Upper Layer Bindings . . . . . . . . . . . 11 73 8. Security Considerations . . . . . . . . . . . . . . . . . . . 12 74 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 75 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 76 11. Normative References . . . . . . . . . . . . . . . . . . . . 13 77 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 79 1. Introduction 81 The purpose of this document is to enable concurrent operation in 82 both directions on a single transport connection using RPC-over-RDMA 83 protocol versions that do not have specific facilities for backward 84 direction operation. 86 Backward direction RPC transactions are necessary for the operation 87 of NFSv4.1, and in particular, of pNFS, though any Upper Layer 88 Protocol implementation may make use of them. An Upper Layer Binding 89 for NFSv4.x callback operation is additionally required (see 90 Section 7), but is not provided in this document. 92 For example, using the approach described herein, RPC transactions 93 can be conveyed in both directions on the same RPC-over-RDMA Version 94 One connection without changes to the the XDR description of RPC- 95 over-RDMA Version One. This document does not modify the XDR or 96 protocol described in [I-D.ietf-nfsv4-rfc5666bis]. Future versions 97 of RPC-over-RDMA may adopt the approach described herein, or may 98 replace it with a different approach. 100 1.1. Requirements Language 102 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 103 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 104 document are to be interpreted as described in [RFC2119]. 106 2. Understanding RPC Direction 108 The ONC RPC protocol as described in [RFC5531] is architected as a 109 message-passing protocol between one server and one or more clients. 110 ONC RPC transactions are made up of two types of messages. 112 A CALL message, or "Call", requests work. A Call is designated by 113 the value CALL in the message's msg_type field. An arbitrary unique 114 value is placed in the message's xid field. A host that originates a 115 Call is referred to in this document as a "Requester." 117 A REPLY message, or "Reply", reports the results of work requested by 118 a Call. A Reply is designated by the value REPLY in the message's 119 msg_type field. The value contained in the message's xid field is 120 copied from the Call whose results are being returned. A host that 121 emits a Reply is referred to as a "Responder." 123 Typically, a Call results in a corresponding Reply. A Reply is never 124 sent without a corresponding Call. 126 RPC-over-RDMA is a connection-oriented RPC transport. In all cases, 127 when a connection-oriented transport is used, ONC RPC client 128 endpoints are responsible for initiating transport connections, while 129 ONC RPC service endpoints passively await incoming connection 130 requests. 132 RPC direction on connectionless RPC transports is not addressed in 133 this document. 135 2.1. Forward Direction 137 Traditionally, an ONC RPC client acts as a Requester, while an ONC 138 RPC service acts as a Responder. This form of message passing is 139 referred to as "forward direction" operation. 141 2.2. Backward Direction 143 The ONC RPC specification [RFC5531] does not forbid passing messages 144 in the other direction. An ONC RPC service endpoint can act as a 145 Requester, in which case an ONC RPC client endpoint acts as a 146 Responder. This form of message passing is referred to as "backward 147 direction" operation. 149 During backward direction operation, the ONC RPC client is 150 responsible for establishing transport connections, even though ONC 151 RPC Calls come from the ONC RPC server. 153 ONC RPC clients and services are optimized to perform and scale well 154 while handling traffic in the forward direction, and may not be 155 prepared to handle operation in the backward direction. Not until 156 NFSv4.1 [RFC5661] has there been a strong need to handle backward 157 direction operation. 159 2.3. Bi-directional Operation 161 A pair of connected RPC endpoints may choose to use only forward or 162 only backward direction operations on a particular transport. Or, 163 these endpoints may send Calls in both directions concurrently on the 164 same transport. 166 "Bi-directional operation" occurs when both transport endpoints act 167 as a Requester and a Responder at the same time. 169 Bi-directionality is an extension of RPC transport connection 170 sharing. Two RPC endpoints wish to exchange independent RPC messages 171 over a shared connection, but in opposite directions. These messages 172 may or may not be related to the same workloads or RPC Programs. 174 2.4. XID Values 176 Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, 177 or "xid" for short. The value of an xid is interpreted in the 178 context of the message's msg_type field. 180 o The xid of a Call is arbitrary but is unique among outstanding 181 Calls from that Requester. 183 o The xid of a Reply always matches that of the initiating Call. 185 When receiving a Reply, a Requester matches the xid value in the 186 Reply with a Call it previously sent. 188 2.4.1. XID Generation 190 During bi-directional operation, forward and backward direction XIDs 191 are typically generated on distinct hosts by possibly different 192 algorithms. There is no co-ordination between forward and backward 193 direction XID generation. 195 Therefore, a forward direction Requester MAY use the same xid value 196 at the same time as a backward direction Requester on the same 197 transport connection. Though such concurrent requests use the same 198 xid value, they represent distinct ONC RPC transactions. 200 3. Immediate Uses Of Bi-Directional RPC-over-RDMA 202 3.1. NFSv4.0 Callback Operation 204 An NFSv4.0 client employs a traditional ONC RPC client to send NFS 205 requests to an NFSv4.0 server's traditional ONC RPC service 206 [RFC7530]. NFSv4.0 requests flow in the forward direction on a 207 connection established by the client. This connection is referred to 208 as a "forechannel" connection. 210 An NFSv4 "delegation" is simply a promise made by a server that it 211 will notify a client before another client or program running on the 212 server is allowed access to a file. With this guarantee, that client 213 can operate as sole accessor of the file. In particular, it can 214 manage the file's data and metadata caches aggressively. 216 To administer file delegations, NFSv4.0 introduces the use of 217 callback operations, or "callbacks", in Section 10.2 of [RFC7530]. 218 An NFSv4.0 server sets up a traditional ONC RPC client, and an 219 NFSv4.0 client sets up a traditional ONC RPC service. Callbacks flow 220 in the forward direction on a connection established between the 221 server's callback client, and the client's callback server. This 222 connection is distinct from connections being used as forechannels, 223 and is referred to as a "backchannel connection." 225 When an RDMA transport is used as a forechannel, an NFSv4.0 client 226 typically provides a TCP callback service. The client's SETCLIENTID 227 operation advertises the callback service endpoint with a "tcp" or 228 "tcp6" netid. The server then connects to this service using a TCP 229 socket. 231 NFSv4.0 implementations can function without a backchannel in place. 232 In this case, the server does not grant file delegations. This might 233 result in a negative performance effect, but correctness is not 234 affected. 236 3.2. NFSv4.1 Callback Operation 238 NFSv4.1 supports file delegation in a similar fashion to NFSv4.0, and 239 extends the callback mechanism to manage pNFS layouts, as discussed 240 in Section 12 of [RFC5661]. 242 To facilitate operation through NAT routers, all NFSv4.1 transport 243 connections are initiated by NFSv4.1 clients. Therefore NFSv4.1 244 servers send callbacks to clients in the backward direction on 245 connections established by NFSv4.1 clients. 247 NFSv4.1 clients and servers indicate to their peers that a 248 backchannel capability is available on a given transport in the 249 arguments and results of NFS CREATE_SESSION or BIND_CONN_TO_SESSION 250 operations. 252 NFSv4.1 clients may establish distinct transport connections for 253 forechannel and backchannel operation, or they may combine 254 forechannel and backchannel operation on one transport connection 255 using bi-directional operation. 257 Without a backward direction RPC-over-RDMA capability, an NFSv4.1 258 client must additionally connect using a transport with backward 259 direction capability to use as a backchannel. TCP is the only choice 260 for an NFSv4.1 backchannel connection in this case. 262 Implementations often find it more convenient to use a single 263 combined transport (ie. a transport that is capable of bi-directional 264 operation). This simplifies connection establishment and recovery 265 during network partitions or when one endpoint restarts. This can 266 also enable better scaling by using fewer transport connections to 267 perform the same work. 269 As with NFSv4.0, if a backchannel is not in use, an NFSv4.1 server 270 does not grant delegations. Because NFSv4.1 relies on callbacks to 271 manage pNFS layout state, pNFS operation is not possible without a 272 backchannel. 274 4. Flow Control 276 For an RDMA Send operation to work properly, the receiving peer must 277 have posted a receive buffer in which to accept the incoming message. 278 If a receiver hasn't posted enough buffers to accommodate each 279 incoming Send operation, the receiving RDMA provider is allowed to 280 terminate the RDMA connection. 282 RPC-over-RDMA transport protocols provide built-in send flow control 283 to prevent overrunning the number of pre-posted receive buffers on a 284 connection's receive endpoint. For RPC-over-RDMA Version One, this 285 is discussed in Section 4.3 of [I-D.ietf-nfsv4-rfc5666bis]. 287 4.1. Backward Credits 289 RPC-over-RDMA credits work the same way in the backward direction as 290 they do in the forward direction. However, forward direction credits 291 and backward direction credits on the same connection are accounted 292 separately. 294 The forward direction credit value retains the same meaning whether 295 or not there are backward direction resources associated with an RPC- 296 over-RDMA transport connection. This is the number of RPC requests 297 the forward direction responder (the RPC server) is prepared to 298 receive concurrently. 300 The backward direction credit value is the number of RPC requests the 301 backward direction responder (the RPC client) is prepared to receive 302 concurrently. The backward direction credit value MAY be different 303 than the forward direction credit value. 305 During bi-directional operation, each receiver has to decide whether 306 an incoming message contains a credit request (the receiver is acting 307 as a responder) or a credit grant (the receiver is acting as a 308 requester) and apply the credit value accordingly. 310 When message direction is not fully determined by context (e.g., 311 suggested by the definition of the RPC-over-RDMA version that is in 312 use) or by an accompanying RPC message payload with a call direction 313 field, it is not possible for the receiver to tell with certainty 314 whether the header credit value is a request or grant. In such 315 cases, the receiver MUST NOT use the header's credit value. 317 4.2. Inline Thresholds 319 Forward and backward operation on the same connection share the same 320 receive buffers. Therefore the inline threshold values for the 321 forward direction and the backward direction are the same. The call 322 inline threshold for the backward direction is the same as the reply 323 inline threshold for the forward direction, and vice versa. For more 324 information, see Section 4.3.2 of [I-D.ietf-nfsv4-rfc5666bis]. 326 4.3. Managing Receive Buffers 328 An RPC-over-RDMA transport endpoint must pre-post receive buffers 329 before it can receive and process incoming RPC-over-RDMA messages. 330 If a sender transmits a message for a receiver which has no posted 331 receive buffer, the RDMA provider is allowed to drop the RDMA 332 connection. 334 4.3.1. Client Receive Buffers 336 Typically an RPC-over-RDMA Requester posts only as many receive 337 buffers as there are outstanding RPC Calls. A client endpoint 338 without backward direction support might therefore at times have no 339 pre-posted receive buffers. 341 To receive incoming backward direction Calls, an RPC-over-RDMA client 342 endpoint must pre-post enough additional receive buffers to match its 343 advertised backward direction credit value. Each outstanding forward 344 direction RPC requires an additional receive buffer above this 345 minimum. 347 When an RDMA transport connection is lost, all active receive buffers 348 are flushed and are no longer available to receive incoming messages. 349 When a fresh transport connection is established, a client endpoint 350 must re-post a receive buffer to handle the Reply for each 351 retransmitted forward direction Call, and a full set of receive 352 buffers to handle backward direction Calls. 354 4.3.2. Server Receive Buffers 356 A forward direction RPC-over-RDMA service endpoint posts as many 357 receive buffers as it expects incoming forward direction Calls. That 358 is, it posts no fewer buffers than the number of credits granted in 359 the rdma_credit field of forward direction RPC replies. 361 To receive incoming backward direction replies, an RPC-over-RDMA 362 server endpoint must pre-post a receive buffer for each backward 363 direction Call it sends. 365 When the existing transport connection is lost, all active receive 366 buffers are flushed and are no longer available to receive incoming 367 messages. When a fresh transport connection is established, a server 368 endpoint must re-post a receive buffer to handle the Reply for each 369 retransmitted backward direction Call, and a full set of receive 370 buffers for receiving forward direction Calls. 372 5. Sending And Receiving Backward Operations 374 The operation of RPC-over-RDMA transports in the forward direction is 375 defined in [RFC5531] and [I-D.ietf-nfsv4-rfc5666bis]. In this 376 section, a mechanism for backward direction operation on RPC-over- 377 RDMA is defined. Backward operation used in combination with forward 378 operation enables bi-directional communication on a common RPC 379 transport connection. 381 Certain fields in the RPC-over-RDMA header are fixed for all versions 382 of RPC-over-RDMA. The XDR description of these fields is contained 383 in Section 5.1 of [I-D.ietf-nfsv4-rfc5666bis]. 385 5.1. Sending A Backward Direction Call 387 To form a backward direction RPC-over-RDMA Call message, an ONC RPC 388 service endpoint constructs an RPC-over-RDMA header containing a 389 fresh RPC XID in the rdma_xid field (see Section 2.4 for full 390 requirements). 392 The rdma_vers field MUST contain the same value in backward and 393 forward direction Call messages on the same connection. 395 The number of requested backward direction credits is placed in the 396 rdma_credit field (see Section 4). 398 Whether presented inline or as a separate chunk, the ONC RPC Call 399 header MUST start with the same XID value that is present in the RPC- 400 over-RDMA header, and the RPC header's msg_type field MUST contain 401 the value CALL. 403 5.2. Sending A Backward Direction Reply 405 To form a backward direction RPC-over-RDMA Reply message, an ONC RPC 406 client endpoint constructs an RPC-over-RDMA header containing a copy 407 of the matching ONC RPC Call's RPC XID in the rdma_xid field (see 408 Section 2.4 for full requirements). 410 The rdma_vers field MUST contain the same value in a backward 411 direction Reply message as in the matching Call message. 413 The number of granted backward direction credits is placed in the 414 rdma_credit field (see Section 4). 416 Whether presented inline or as a separate chunk, the ONC RPC Reply 417 header MUST start with the same XID value that is present in the RPC- 418 over-RDMA header, and the RPC header's msg_type field MUST contain 419 the value REPLY. 421 5.3. Backward Direction Chunks 423 Chunks MAY be used in the backward direction. They operate the same 424 way as in the forward direction (see [I-D.ietf-nfsv4-rfc5666bis] for 425 details). 427 An implementation might not support any Upper Layer Protocol that has 428 DDP-eligible data items. The Upper Layer Protocol may also use only 429 small messages, or it may have a native mechanism for restricting the 430 size of backward direction RPC messages, obviating the need to handle 431 Long Messages in the backward direction. 433 When there is no Upper Layer Protocol requirement for chunks, 434 implementers can choose not to provide support for chunks in the 435 backward direction. This avoids the complexity of adding support for 436 performing RDMA Reads and Writes in the backward direction. 438 When chunks are not implemented, RPC messages in the backward 439 direction are always sent using RDMA_MSG, and therefore can be no 440 larger than what can be sent inline (that is, without chunks). 441 Sending an inline message larger than the inline threshold can result 442 in loss of connection. 444 If a backward direction requester provides a non-empty chunk list to 445 a responder that does not support chunks, the responder MUST reply 446 with an RDMA_ERROR message with rdma_err field set to ERR_CHUNK. 448 5.4. Backward Direction Retransmission 450 In rare cases, an ONC RPC transaction cannot be completed within a 451 certain time. This can be because the transport connection was lost, 452 the Call or Reply message was dropped, or because the Upper Layer 453 consumer delayed or dropped the ONC RPC request. Typically, the 454 Requester sends the transaction again, reusing the same RPC XID. 455 This is known as an "RPC retransmission". 457 In the forward direction, the Requester is the ONC RPC client. The 458 client is always responsible for establishing a transport connection 459 before sending again. 461 In the backward direction, the Requester is the ONC RPC server. 462 Because an ONC RPC server does not establish transport connections 463 with clients, it cannot send a retransmission if there is no 464 transport connection. It must wait for the ONC RPC client to re- 465 establish the transport connection before it can retransmit ONC RPC 466 transactions in the backward direction. 468 If an ONC RPC client has no work to do, it may be some time before it 469 re-establishes a transport connection. Backward direction Requesters 470 must be prepared to wait indefinitely for a connection to be 471 established before a pending backward direction ONC RPC Call can be 472 retransmitted. 474 Forward direction Requesters are responsible for maintaining a 475 transport connection as long as there is the possibility of backward 476 direction requests. For example, an NFSv4.1 client with open 477 delegated files or active pNFS layouts should maintain a transport 478 connection so the server can send callback operations. 480 6. In the Absence of Backward Direction Support 482 An RPC-over-RDMA transport endpoint might not support backward 483 direction operation (and thus bi-directional operation). There might 484 be no mechanism in the transport implementation to do so. Or in an 485 implementation that can support operation in the backward direction, 486 the Upper Layer Protocol consumer might not yet have configured or 487 enabled the transport to handle backward direction traffic. 489 If an endpoint is not prepared to receive an incoming backward 490 direction message, loss of the RDMA connection might result. Thus 491 denial of service could result if a sender continues to send backward 492 direction messages after every transport reconnect to an endpoint 493 that is not prepared to receive them. 495 When dealing with the possibility that the remote peer has no 496 transport level support for backward direction operation, the Upper 497 Layer Protocol becomes responsible for informing peers when backward 498 direction operation is supported. Otherwise even a simple backward 499 direction NULL probe from a peer could result in a lost connection. 501 Therefore, an Upper Layer Protocol consumer MUST NOT perform backward 502 direction ONC RPC operations unless the peer consumer has indicated 503 it is prepared to handle them. A description of Upper Layer Protocol 504 mechanisms used for this indication is outside the scope of this 505 document. 507 For example, an NFSv4.1 server does not send backchannel messages to 508 an NFSv4.1 client before the NFSv4.1 client has sent a CREATE_SESSION 509 or a BIND_CONN_TO_SESSION operation. As long as an NFSv4.1 client 510 has prepared appropriate backchannel resources before sending one of 511 these operations announcing support for backchannel operation, denial 512 of service is avoided. 514 7. Considerations For Upper Layer Bindings 516 An Upper Layer Protocol that operates on RPC-over-RDMA transports may 517 have procedures that include DDP-eligible data items. DDP- 518 eligibility is specified in an Upper Layer Binding. Direction of 519 operation does not obviate the need for DDP-eligibility statements. 521 Backward-only operation requires the client endpoint to establish a 522 fresh connection. The Upper Layer Binding can specify appropriate 523 RPC binding parameters for such connections. 525 Bi-directional operation occurs on an already-established connection. 526 Specification of RPC binding parameters is usually not necessary in 527 this case. 529 For bi-directional operation, other considerations about sharing an 530 RPC-over-RDMA transport with another ULP may apply. Consult 531 Section 7 of [I-D.ietf-nfsv4-rfc5666bis] for details about what else 532 may be contained in an Upper Layer Binding. 534 8. Security Considerations 536 Security considerations for operation on RPC-over-RDMA transports are 537 outlined in Section 9 of [I-D.ietf-nfsv4-rfc5666bis]. 539 9. IANA Considerations 541 This document does not require actions by IANA. 543 10. Acknowledgements 545 Tom Talpey was an indispensable resource, in addition to creating the 546 foundation upon which this work is based. Our warmest regards go to 547 him for his help and support. 549 Dave Noveck provided excellent review, constructive suggestions, and 550 navigational guidance throughout the process of drafting this 551 document. 553 Dai Ngo was a solid partner and collaborator. Together we 554 constructed and tested independent prototypes of the changes 555 described in this document. 557 The author wishes to thank Bill Baker for his unwavering support of 558 this work. In addition, the author gratefully acknowledges the 559 expert contributions of Karen Deitke, Chunli Zhang, Mahesh 560 Siddheshwar, Steve Wise, and Tom Tucker. 562 Special thanks go to the nfsv4 Working Group Chair Spencer Shepler 563 and the nfsv4 Working Group Secretary Tom Haynes for their support. 565 11. Normative References 567 [I-D.ietf-nfsv4-rfc5666bis] 568 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 569 Memory Access Transport for Remote Procedure Call", draft- 570 ietf-nfsv4-rfc5666bis-04 (work in progress), March 2016. 572 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 573 Requirement Levels", BCP 14, RFC 2119, March 1997. 575 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 576 Specification Version 2", RFC 5531, May 2009. 578 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 579 System (NFS) Version 4 Minor Version 1 Protocol", RFC 580 5661, January 2010. 582 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 583 Version 4 Protocol", RFC 7530, March 2015. 585 Author's Address 587 Charles Lever 588 Oracle Corporation 589 1015 Granger Avenue 590 Ann Arbor, MI 48104 591 USA 593 Phone: +1 734 274 2396 594 Email: chuck.lever@oracle.com