idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-bidirection-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 8, 2017) is 2578 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-10 ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Standards Track March 8, 2017 5 Expires: September 9, 2017 7 Bi-directional Remote Procedure Call On RPC-over-RDMA Transports 8 draft-ietf-nfsv4-rpcrdma-bidirection-08 10 Abstract 12 Minor versions of Network File System (NFS) version 4 newer than 13 minor version 0 work best when Remote Procedure Call (RPC) transports 14 can send RPC transactions in both directions on the same connection. 15 This document describes how RPC transport endpoints capable of Remote 16 Direct Memory Access (RDMA) convey RPCs in both directions on a 17 single connection. 19 Requirements Language 21 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 22 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 23 document are to be interpreted as described in [RFC2119]. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on September 9, 2017. 42 Copyright Notice 44 Copyright (c) 2017 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 60 2. Understanding RPC Direction . . . . . . . . . . . . . . . . . 3 61 3. Immediate Uses Of Bi-Directional RPC-over-RDMA . . . . . . . 5 62 4. Flow Control . . . . . . . . . . . . . . . . . . . . . . . . 6 63 5. Sending And Receiving Operations In The Reverse Direction . . 8 64 6. In the Absence of Support For Reverse Direction Operation . . 11 65 7. Considerations For Upper Layer Bindings . . . . . . . . . . . 12 66 8. Security Considerations . . . . . . . . . . . . . . . . . . . 12 67 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 68 10. Normative References . . . . . . . . . . . . . . . . . . . . 12 69 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 13 70 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 72 1. Introduction 74 RPC-over-RDMA transports, introduced in [I-D.ietf-nfsv4-rfc5666bis], 75 efficiently convey Remote Procedure Call transactions (RPCs) on 76 transport layers capable of Remote Direct Memory Access (RDMA). The 77 purpose of this document is to enable concurrent operation in both 78 directions on a single transport connection using RPC-over-RDMA 79 protocol versions that do not have specific facilities for reverse 80 direction operation. 82 Reverse direction RPC transactions are necessary for the operation of 83 version 4.1 of the Network File System (NFS), and in particular, of 84 Parallel NFS (pNFS) [RFC5661], though any Upper Layer Protocol 85 implementation may make use of them. An Upper Layer Binding for NFS 86 version 4.x callback operation is additionally required (see 87 Section 7), but is not provided in this document. 89 For example, using the approach described herein, RPC transactions 90 can be conveyed in both directions on the same RPC-over-RDMA Version 91 One connection without changes to the RPC-over-RDMA Version One 92 protocol. This document does not update the protocol specified in 93 [I-D.ietf-nfsv4-rfc5666bis]. 95 The remainder of this document assumes familiarity with the 96 terminology and concepts contained in [I-D.ietf-nfsv4-rfc5666bis], 97 especially Sections 2 and 3. 99 2. Understanding RPC Direction 101 The Remote Procedure Call (ONC RPC) protocol as described in 102 [RFC5531] is architected as a message-passing protocol between one 103 server and one or more clients. ONC RPC transactions are made up of 104 two types of messages. 106 A CALL message, or "Call", requests work. A Call is designated by 107 the value CALL in the message's msg_type field. An arbitrary unique 108 value is placed in the message's XID field. A host that originates a 109 Call is referred to in this document as a "Requester." 111 A REPLY message, or "Reply", reports the results of work requested by 112 a Call. A Reply is designated by the value REPLY in the message's 113 msg_type field. The value contained in the message's XID field is 114 copied from the Call whose results are being returned. A host that 115 emits a Reply is referred to as a "Responder." 117 Typically, a Call results in a corresponding Reply. A Reply is never 118 sent without a corresponding Call. 120 RPC-over-RDMA is a connection-oriented RPC transport. In all cases, 121 when a connection-oriented transport is used, ONC RPC client 122 endpoints are responsible for initiating transport connections, while 123 ONC RPC service endpoints passively await incoming connection 124 requests. 126 RPC direction on connectionless RPC transports is not addressed in 127 this document. 129 2.1. Forward Direction 131 Traditionally, an ONC RPC client acts as a Requester, while an ONC 132 RPC service acts as a Responder. This form of message passing is 133 referred to as "forward direction" operation. 135 2.2. Reverse Direction 137 The ONC RPC specification [RFC5531] does not forbid passing messages 138 in the other direction. An ONC RPC service endpoint can act as a 139 Requester, in which case an ONC RPC client endpoint acts as a 140 Responder. This form of message passing is referred to as "reverse 141 direction" operation. 143 During reverse direction operation, the ONC RPC client is responsible 144 for establishing transport connections, even though ONC RPC Calls 145 come from the ONC RPC server. 147 ONC RPC clients and servers are optimized to perform and scale well 148 while handling traffic in the forward direction, and might not be 149 prepared to handle operation in the reverse direction. Not until NFS 150 version 4.1 [RFC5661] has there been a strong need to handle reverse 151 direction operation. 153 2.3. Bi-directional Operation 155 A pair of connected RPC endpoints may choose to use only forward or 156 only reverse direction operations on a particular transport. Or, 157 these endpoints may send Calls in both directions concurrently on the 158 same transport. 160 "Bi-directional operation" occurs when both transport endpoints act 161 as a Requester and a Responder at the same time. 163 Bi-directionality is an extension of RPC transport connection 164 sharing. Two RPC endpoints wish to exchange independent RPC messages 165 over a shared connection, but in opposite directions. These messages 166 may or may not be related to the same workloads or RPC Programs. 168 2.4. XID Values 170 Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, 171 or "XID" for short. The value of an XID is interpreted in the 172 context of the message's msg_type field. 174 o The XID of a Call is arbitrary but is unique among outstanding 175 Calls from that Requester. 177 o The XID of a Reply always matches that of the initiating Call. 179 When receiving a Reply, a Requester matches the XID value in the 180 Reply with a Call it previously sent. 182 2.4.1. XID Generation 184 During bi-directional operation, forward and reverse direction XIDs 185 are typically generated on distinct hosts by possibly different 186 algorithms. There is no co-ordination between forward and reverse 187 direction XID generation. 189 Therefore, a forward direction Requester MAY use the same XID value 190 at the same time as a reverse direction Requester on the same 191 transport connection. Though such concurrent requests use the same 192 XID value, they represent distinct ONC RPC transactions. 194 3. Immediate Uses Of Bi-Directional RPC-over-RDMA 196 3.1. NFS version 4.0 Callback Operation 198 An NFS version 4.0 client employs a traditional ONC RPC client to 199 send NFS requests to an NFS version 4.0 server's traditional ONC RPC 200 service [RFC7530]. NFS version 4.0 requests flow in the forward 201 direction on a connection established by the client. This connection 202 is referred to as a "forechannel" connection. 204 An NFS version 4.x "delegation" is simply a promise made by a server 205 that it will notify a client before another client or program running 206 on the server is allowed access to a file. With this guarantee, that 207 client can operate as sole accessor of the file. In particular, it 208 can manage the file's data and metadata caches aggressively. 210 To administer file delegations, NFS version 4.0 introduces the use of 211 callback operations, or "callbacks", in Section 10.2 of [RFC7530]. 212 An NFS version 4.0 server sets up a forward direction ONC RPC client, 213 and an NFS version 4.0 client sets up a forward direction ONC RPC 214 service. Callbacks flow in the forward direction on a connection 215 established between the server's callback client, and the client's 216 callback service. This connection is distinct from connections being 217 used as forechannels, and is referred to as a "backchannel 218 connection." 220 When an RDMA transport is used as a forechannel, an NFS version 4.0 221 client typically provides a TCP-based callback service. The client's 222 SETCLIENTID operation advertises the callback service endpoint with a 223 "tcp" or "tcp6" netid. The server then connects to this service 224 using a TCP socket. 226 NFS version 4.0 implementations can function without a backchannel in 227 place. In this case, the NFS server does not grant file delegations. 228 This might result in a negative performance effect, but correctness 229 is not affected. 231 3.2. NFS version 4.1 Callback Operation 233 NFS version 4.1 supports file delegation in a similar fashion to NFS 234 version 4.0, and extends the callback mechanism to manage pNFS 235 layouts, as discussed in Section 12 of [RFC5661]. 237 NFS version 4.1 transport connections are initiated by NFS version 238 4.1 clients. Therefore NFS version 4.1 servers send callbacks to 239 clients in the reverse direction on connections established by NFS 240 version 4.1 clients. 242 NFS version 4.1 clients and servers indicate to their peers that a 243 backchannel capability is available on a given transport in the 244 arguments and results of the NFS CREATE_SESSION or 245 BIND_CONN_TO_SESSION operations. 247 NFS version 4.1 clients may establish distinct transport connections 248 for forechannel and backchannel operation, or they may combine 249 forechannel and backchannel operation on one transport connection 250 using bi-directional operation. 252 Without a reverse direction RPC-over-RDMA capability, an NFS version 253 4.1 client additionally connects using a transport with reverse 254 direction capability to use as a backchannel. Opening an independent 255 TCP socket is the only choice for an NFS version 4.1 backchannel 256 connection in this case. 258 Implementations often find it more convenient to use a single 259 combined transport (i.e. a transport that is capable of bi- 260 directional operation). This simplifies connection establishment and 261 recovery during network partitions or when one endpoint restarts. 262 This can also enable better scaling by using fewer transport 263 connections to perform the same work. 265 As with NFS version 4.0, if a backchannel is not in use, an NFS 266 version 4.1 server does not grant delegations. Because NFS version 267 4.1 relies on callbacks to manage pNFS layout state, pNFS operation 268 is not possible without a backchannel. 270 4. Flow Control 272 For an RDMA Send operation to work properly, the receiving peer has 273 to have already posted a receive buffer in which to accept the 274 incoming message. If a receiver hasn't posted enough buffers to 275 accommodate each incoming Send operation, the receiving RDMA provider 276 is allowed to terminate the RDMA connection. 278 RPC-over-RDMA transport protocols provide built-in send flow control 279 to prevent overrunning the number of pre-posted receive buffers on a 280 connection's receive endpoint using a "credit grant" mechanism. The 281 use of credits in RPC-over-RDMA Version One is described in 282 Section 3.3 of [I-D.ietf-nfsv4-rfc5666bis]. 284 4.1. Reverse-direction Credits 286 RPC-over-RDMA credits work the same way in the reverse direction as 287 they do in the forward direction. However, forward direction credits 288 and reverse direction credits on the same connection are accounted 289 separately. Direction-independent credit accounting prevents head- 290 of-line blocking in one direction from impacting operation in the 291 other direction. 293 The forward direction credit value retains the same meaning whether 294 or not there are reverse direction resources associated with an RPC- 295 over-RDMA transport connection. This is the number of RPC requests 296 the forward direction responder (the ONC RPC server) is prepared to 297 receive concurrently. 299 The reverse direction credit value is the number of RPC requests the 300 reverse direction responder (the ONC RPC client) is prepared to 301 receive concurrently. The reverse direction credit value MAY be 302 different than the forward direction credit value. 304 During bi-directional operation, each receiver has to decide whether 305 an incoming message contains a credit request (the receiver is acting 306 as a responder) or a credit grant (the receiver is acting as a 307 requester) and apply the credit value accordingly. 309 When message direction is not fully determined by context (e.g., 310 suggested by the definition of the RPC-over-RDMA version that is in 311 use) or by an accompanying RPC message payload with a call direction 312 field, it is not possible for the receiver to tell with certainty 313 whether the header credit value is a request or grant. In such 314 cases, the receiver MUST ignore the header's credit value. 316 4.2. Inline Thresholds 318 Forward and reverse direction operation on the same connection share 319 the same receive buffers. Therefore the inline threshold values for 320 the forward direction and the reverse direction are the same. The 321 call inline threshold for the reverse direction is the same as the 322 reply inline threshold for the forward direction, and vice versa. 323 For more information, see Section 3.3.2 of 324 [I-D.ietf-nfsv4-rfc5666bis]. 326 4.3. Managing Receive Buffers 328 An RPC-over-RDMA transport endpoint posts receive buffers before it 329 can receive and process incoming RPC-over-RDMA messages. If a sender 330 transmits a message for a receiver which has no posted receive 331 buffer, the RDMA provider is allowed to drop the RDMA connection. 333 4.3.1. Client Receive Buffers 335 Typically an RPC-over-RDMA Requester posts only as many receive 336 buffers as there are outstanding RPC Calls. A client endpoint 337 without reverse direction support might therefore at times have no 338 available receive buffers. 340 To receive incoming reverse direction Calls, an RPC-over-RDMA client 341 endpoint posts enough additional receive buffers to match its 342 advertised reverse direction credit value. Each outstanding forward 343 direction RPC requires an additional receive buffer above this 344 minimum. 346 When an RDMA transport connection is lost, all active receive buffers 347 are flushed and are no longer available to receive incoming messages. 348 When a fresh transport connection is established, a client endpoint 349 re-posts a receive buffer to handle the Reply for each retransmitted 350 forward direction Call, and a full set of receive buffers to handle 351 reverse direction Calls. 353 4.3.2. Server Receive Buffers 355 A forward direction RPC-over-RDMA service endpoint posts as many 356 receive buffers as it expects incoming forward direction Calls. That 357 is, it posts no fewer buffers than the number of credits granted in 358 the rdma_credit field of forward direction RPC replies. 360 To receive incoming reverse direction replies, an RPC-over-RDMA 361 server endpoint posts enough additional receive buffers to handle 362 replies for each reverse direction Call it sends. 364 When the existing transport connection is lost, all active receive 365 buffers are flushed and are no longer available to receive incoming 366 messages. When a fresh transport connection is established, a server 367 endpoint re-posts a receive buffer to handle the Reply for each 368 retransmitted reverse direction Call, and a full set of receive 369 buffers for receiving forward direction Calls. 371 5. Sending And Receiving Operations In The Reverse Direction 373 The operation of RPC-over-RDMA transports in the forward direction is 374 defined in [RFC5531] and [I-D.ietf-nfsv4-rfc5666bis]. In this 375 section, a mechanism for reverse direction operation on RPC-over-RDMA 376 is defined. Reverse direction operation used in combination with 377 forward operation enables bi-directional communication on a common 378 RPC-over-RDMA transport connection. 380 Certain fields in the RPC-over-RDMA header have a fixed position in 381 all versions of RPC-over-RDMA. The normative specification of these 382 fields is contained in Section 4 of [I-D.ietf-nfsv4-rfc5666bis]. 384 5.1. Sending A Call In The Reverse Direction 386 To form a reverse direction RPC-over-RDMA Call message, an ONC RPC 387 service endpoint constructs an RPC-over-RDMA header containing a 388 fresh RPC XID in the rdma_xid field (see Section 2.4 for full 389 requirements). 391 The rdma_vers field MUST contain the same value in reverse and 392 forward direction Call messages on the same connection. 394 The number of requested reverse direction credits is placed in the 395 rdma_credit field (see Section 4). 397 Whether presented inline or as a separate chunk, the ONC RPC Call 398 header MUST start with the same XID value that is present in the RPC- 399 over-RDMA header, and the RPC header's msg_type field MUST contain 400 the value CALL. 402 5.2. Sending A Reply In The Reverse Direction 404 To form a reverse direction RPC-over-RDMA Reply message, an ONC RPC 405 client endpoint constructs an RPC-over-RDMA header containing a copy 406 of the matching ONC RPC Call's RPC XID in the rdma_xid field (see 407 Section 2.4 for full requirements). 409 The rdma_vers field MUST contain the same value in a reverse 410 direction Reply message as in the matching Call message. 412 The number of granted reverse direction credits is placed in the 413 rdma_credit field (see Section 4). 415 Whether presented inline or as a separate chunk, the ONC RPC Reply 416 header MUST start with the same XID value that is present in the RPC- 417 over-RDMA header, and the RPC header's msg_type field MUST contain 418 the value REPLY. 420 5.3. Using Chunks In Reverse Direction Operations 422 A "chunk" refers to a DDP-eligible portion of a message's Payload 423 stream that is placed directly in the receiver's memory by the 424 transport. Chunk data may be moved by an explicit RDMA operation, 425 for example. Chunks are defined in Section 3.4.4 and DDP-eligibility 426 is covered in Section 6.1 of [I-D.ietf-nfsv4-rfc5666bis]. 428 Chunks MAY be used in the reverse direction. They operate the same 429 way as in the forward direction. 431 An implementation might support only Upper Layer Protocols that have 432 no DDP-eligible data items. Such Upper Layer Protocols may use only 433 small messages, or they may have a native mechanism for restricting 434 the size of reverse direction RPC messages, obviating the need to 435 handle Long Messages in the reverse direction. 437 When there is no Upper Layer Protocol requirement for chunks in the 438 reverse direction, implementers can choose not to provide support for 439 chunks in the reverse direction. This avoids the complexity of 440 adding support for performing RDMA Reads and Writes in the reverse 441 direction. 443 When chunks are not implemented, RPC messages in the reverse 444 direction are always sent using a Short message, and therefore can be 445 no larger than what can be sent inline (that is, without chunks). 446 Sending an inline message larger than the inline threshold can result 447 in loss of connection. 449 If a reverse direction requester provides a non-empty chunk list to a 450 responder that does not support chunks, the responder MUST reply with 451 an RDMA_ERROR message with rdma_err field set to ERR_CHUNK. 453 5.4. Reverse Direction Retransmission 455 In rare cases, an ONC RPC service cannot complete an RPC transaction 456 and then send a reply. This can be because the transport connection 457 was lost, the Call or Reply message was dropped, or because the Upper 458 Layer consumer delayed or dropped the ONC RPC request. Typically, 459 the Requester sends the RPC transaction again, reusing the same RPC 460 XID. This is known as an "RPC retransmission". 462 In the forward direction, the Requester is the ONC RPC client. The 463 client is always responsible for establishing a transport connection 464 before sending again. 466 With reverse direction operation, the Requester is the ONC RPC 467 server. Because an ONC RPC server does not establish transport 468 connections with clients, it cannot retransmit if there is no 469 transport connection. It is forced to wait for the ONC RPC client to 470 re-establish a transport connection before it can retransmit ONC RPC 471 transactions in the reverse direction. 473 If the ONC RPC client peer has no work to do, it can be some time 474 before it re-establishes a transport connection. A waiting reverse 475 direction ONC RPC Call may time out to avoid waiting indefinitely for 476 a connection to be established. 478 Therefore forward direction Requesters SHOULD maintain a transport 479 connection as long as there is the possibility that the connection 480 peer can send reverse direction requests. For example, while an NFS 481 version 4.1 client has open delegated files or active pNFS layouts, 482 it maintains one or more transport connections to enable the NFS 483 server to perform callback operations. 485 6. In the Absence of Support For Reverse Direction Operation 487 An RPC-over-RDMA transport endpoint might not support reverse 488 direction operation (and thus it does not support bi-directional 489 operation). There might be no mechanism in the transport 490 implementation to do so. Or in an implementation that can support 491 operation in the reverse direction, the Upper Layer Protocol consumer 492 might not yet have configured or enabled the transport to handle 493 reverse direction traffic. 495 If an endpoint is not prepared to receive an incoming reverse 496 direction message, loss of the RDMA connection might result. Thus 497 denial of service could result if a sender continues to send reverse 498 direction messages after every transport reconnect to an endpoint 499 that is not prepared to receive them. 501 When dealing with the possibility that the remote peer has no 502 transport level support for reverse direction operation, the Upper 503 Layer Protocol becomes responsible for informing peers when reverse 504 direction operation is supported. Otherwise even a simple reverse 505 direction RPC NULL procedure from a peer could result in a lost 506 connection. 508 Therefore, an Upper Layer Protocol consumer MUST NOT perform reverse 509 direction ONC RPC operations until the peer consumer has indicated it 510 is prepared to handle them. A description of Upper Layer Protocol 511 mechanisms used for this indication is outside the scope of this 512 document. 514 For example, an NFS version 4.1 server does not send backchannel 515 messages to an NFS version 4.1 client before the NFS version 4.1 516 client has sent a CREATE_SESSION or a BIND_CONN_TO_SESSION operation. 517 As long as an NFS version 4.1 client has prepared appropriate 518 resources to receive reverse direction operations before sending one 519 of these NFS operations, denial of service is avoided. 521 7. Considerations For Upper Layer Bindings 523 An Upper Layer Protocol that operates on RPC-over-RDMA transports may 524 have procedures that include DDP-eligible data items. DDP- 525 eligibility is specified in an Upper Layer Binding. Direction of 526 operation does not obviate the need for DDP-eligibility statements. 528 Reverse-direction-only operation requires the client endpoint to 529 establish a fresh connection. The Upper Layer Binding can specify 530 appropriate RPC binding parameters for such connections. 532 Bi-directional operation occurs on an already-established connection. 533 Specification of RPC binding parameters is usually not necessary in 534 this case. 536 For bi-directional operation, other considerations may apply when 537 distinct RPC Programs share an RPC-over-RDMA transport connection 538 concurrently. Consult Section 6 of [I-D.ietf-nfsv4-rfc5666bis] for 539 details about what else may be contained in an Upper Layer Binding. 541 8. Security Considerations 543 RPC security is handled in the RPC layer, which is above the 544 transport layer where RPC-over-RDMA operates. 546 Reverse direction operations make use of an authentication mechanism 547 and credentials that are independent of forward direction operation 548 but otherwise operate in the same fashion as outlined in Section 8.2 549 of [I-D.ietf-nfsv4-rfc5666bis]. 551 9. IANA Considerations 553 This document does not require actions by IANA. 555 10. Normative References 557 [I-D.ietf-nfsv4-rfc5666bis] 558 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 559 Memory Access Transport for Remote Procedure Call, Version 560 One", draft-ietf-nfsv4-rfc5666bis-10 (work in progress), 561 February 2017. 563 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 564 Requirement Levels", BCP 14, RFC 2119, March 1997. 566 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 567 Specification Version 2", RFC 5531, May 2009. 569 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 570 System (NFS) Version 4 Minor Version 1 Protocol", 571 RFC 5661, January 2010. 573 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 574 Version 4 Protocol", RFC 7530, March 2015. 576 Appendix A. Acknowledgements 578 Tom Talpey was an indispensable resource, in addition to creating the 579 foundation upon which this work is based. Our warmest regards go to 580 him for his help and support. 582 Dave Noveck provided excellent review, constructive suggestions, and 583 navigational guidance throughout the process of drafting this 584 document. 586 Dai Ngo was a solid partner and collaborator. Together we 587 constructed and tested independent prototypes of the changes 588 described in this document. 590 The author wishes to thank Bill Baker and Greg Marsden for their 591 unwavering support of this work. In addition, the author gratefully 592 acknowledges the expert contributions of Karen Deitke, Chunli Zhang, 593 Mahesh Siddheshwar, Steve Wise, and Tom Tucker. 595 Special thanks go to Transport Area Director Spencer Dawkins, nfsv4 596 Working Group Chair and document shepherd Spencer Shepler, and nfsv4 597 Working Group Secretary Tom Haynes for their support. 599 Author's Address 601 Charles Lever 602 Oracle Corporation 603 1015 Granger Avenue 604 Ann Arbor, MI 48104 605 USA 607 Phone: +1 248 816 6463 608 Email: chuck.lever@oracle.com