idnits 2.17.1 draft-cel-nfsv4-rpcrdma-bidirection-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (April 24, 2015) is 3289 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) ** Obsolete normative reference: RFC 5666 (Obsoleted by RFC 8166) Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Experimental April 24, 2015 5 Expires: October 26, 2015 7 Size-Limited Bi-directional Remote Procedure Call On Remote Direct 8 Memory Access Transports 9 draft-cel-nfsv4-rpcrdma-bidirection-00 11 Abstract 13 Recent minor versions of NFSv4 work best when ONC RPC transports can 14 send ONC RPC calls in both directions. This document describes 15 conventions that enable RPC-over-RDMA version 1 transport endpoints 16 to interoperate when operation in both directions is necessary. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on October 26, 2015. 35 Copyright Notice 37 Copyright (c) 2015 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 2 54 1.2. Scope Of This Document . . . . . . . . . . . . . . . . . 3 55 1.3. Understanding RPC Direction . . . . . . . . . . . . . . . 3 56 1.3.1. Forward Direction . . . . . . . . . . . . . . . . . . 3 57 1.3.2. Backwards Direction . . . . . . . . . . . . . . . . . 4 58 1.3.3. Bi-direction . . . . . . . . . . . . . . . . . . . . 4 59 1.4. Rationale For RPC-over-RDMA Bi-Direction . . . . . . . . 4 60 1.4.1. NFSv4.0 Callback Operation . . . . . . . . . . . . . 4 61 1.4.2. NFSv4.1 Callback Operation . . . . . . . . . . . . . 5 62 1.5. Design Considerations . . . . . . . . . . . . . . . . . . 5 63 1.5.1. Backwards Compatibility . . . . . . . . . . . . . . . 6 64 1.5.2. Performance Impact . . . . . . . . . . . . . . . . . 6 65 1.5.3. Client Complexity . . . . . . . . . . . . . . . . . . 6 66 1.5.4. Payload Size . . . . . . . . . . . . . . . . . . . . 6 67 2. Conventions For Backwards Operation . . . . . . . . . . . . . 7 68 2.1. Flow Control . . . . . . . . . . . . . . . . . . . . . . 7 69 2.1.1. Forwards Credits . . . . . . . . . . . . . . . . . . 7 70 2.1.2. Backwards Credits . . . . . . . . . . . . . . . . . . 8 71 2.2. Managing Receive Buffers . . . . . . . . . . . . . . . . 8 72 2.2.1. Client Receive Buffers . . . . . . . . . . . . . . . 8 73 2.2.2. Server Receive Buffers . . . . . . . . . . . . . . . 9 74 2.2.3. In the Absense of Backwards Direction Support . . . . 9 75 2.3. Backwards Direction Message Size . . . . . . . . . . . . 10 76 2.4. Sending A Backwards Direction Call . . . . . . . . . . . 10 77 2.5. Sending A Backwards Direction Reply . . . . . . . . . . . 11 78 3. Limits To This Approach . . . . . . . . . . . . . . . . . . . 11 79 3.1. Payload Size . . . . . . . . . . . . . . . . . . . . . . 11 80 3.2. Preparedness To Handle Backwards Requests . . . . . . . . 11 81 3.3. Long Term . . . . . . . . . . . . . . . . . . . . . . . . 12 82 4. Security Considerations . . . . . . . . . . . . . . . . . . . 12 83 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 84 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 85 7. Normative References . . . . . . . . . . . . . . . . . . . . 12 86 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 88 1. Introduction 90 1.1. Requirements Language 92 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 93 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 94 document are to be interpreted as described in [RFC2119]. 96 1.2. Scope Of This Document 98 This document describes a set of experimental conventions that apply 99 to RPC-over-RDMA version 1, specified in [RFC5666]. When observed, 100 these conventions enable RPC-over-RDMA version 1 endpoints to handle 101 RPC calls that flow from client to server and server to client 102 concurrently. 104 No changes to the RPC-over-RDMA version 1 protocol definition are 105 needed, thus this document does not update [RFC5666]. 107 The purpose of this document is to permit interoperable prototype 108 implementations of bi-directional RPC-over-RDMA, finally enabling 109 NFSv4.1 (and later NFS minor versions) on RDMA transports. 111 Providing an Upper Layer Binding for NFSv4.x callback operations is 112 not in the scope of this document. 114 1.3. Understanding RPC Direction 116 The ONC RPC protocol, as described in [RFC5531], is fundamentally a 117 message-passing protocol involving one server and perhaps multiple 118 clients. There are two types of messages. 120 A CALL message requests work. A CALL message is designated by the 121 value CALL in the message's msg_type field. An arbitrary unique 122 value is placed in the message's xid field. The host that originates 123 a CALL message is referred to as the "caller." 125 A REPLY message reports the results of requested work. A REPLY 126 message is designated by the value REPLY in the message's msg_type 127 field. The value contained in the message's xid field is copied from 128 the CALL message whose results are being reported. The host that 129 emits a REPLY message is referred to as the "responder." 131 RPC-over-RDMA is a connection-oriented RPC transport. When a 132 connection-oriented transport is used, ONC RPC client endpoints are 133 responsible for initiating transport connections, while ONC RPC 134 service endpoints await passively for incoming connection requests. 135 We do not consider RPC direction on connectionless RPC transports in 136 this document. 138 1.3.1. Forward Direction 140 A traditional ONC RPC client is always the caller. A traditional ONC 141 RPC service is always the responder. This traditional form of ONC 142 RPC message passing is referred to as operation in the "forward 143 direction." 144 During forwards direction operation, the ONC RPC client is 145 responsible for establishing transport connections. 147 1.3.2. Backwards Direction 149 The ONC RPC standard does not forbid passing messages in the other 150 direction. An ONC RPC service endpoint can act as a caller, in which 151 case an ONC RPC client endpoint acts as a responder. This form of 152 message passing is referred to as operation in the "backwards 153 direction." 155 During backwards direction operation, the ONC RPC client is 156 responsible for establishing transport connections, even though ONC 157 RPC calls come from the ONC RPC server. 159 Notably, traditional ONC RPC clients and services are usually not 160 prepared for backwards operation. ONC RPC clients and services are 161 heavily optimized to perform and scale well while handling traffic in 162 the forward direction. Not until recently has there been any need to 163 handle operation in the backwards direction. 165 1.3.3. Bi-direction 167 Finally, bi-directional operation occurs when both transport 168 endpoints act as a caller and a responder at the same time. As 169 above, the ONC RPC client is responsible for establishing transport 170 connections. 172 1.4. Rationale For RPC-over-RDMA Bi-Direction 174 1.4.1. NFSv4.0 Callback Operation 176 An NFSv4.0 client employs a traditional ONC RPC client to send NFS 177 requests to an NFSv4.0 server's traditional ONC RPC service 178 [RFC7530]. NFSv4.0 requests flow in a forward direction on a 179 connection established by the client. This connection is referred to 180 as a "forechannel." 182 NFSv4.0 introduces the use of callback operations, in Section 10.2 of 183 [RFC7530], for managing file delegation. An NFSv4.0 server sets up a 184 traditional ONC RPC client and an NFSv4.0 client sets up a 185 traditional ONC RPC service to handle callback operations. These 186 requests flow in a forward direction on a connection established by 187 the server. This connection is referred to as a "backchannel." 189 When an RDMA transport is used for the forechannel, an NFSv4.0 client 190 typically provides a TCP callback service. The client's SETCLIENTID 191 operation advertises the callback service endpoint with a "tcp" or 192 "tcp6" netid. The server then connects to this service using a TCP 193 socket. 195 NFSv4.0 is fully functional without a backchannel in place. The 196 server simply does not grant file delegations. Applications might 197 experience a performance impact, but operational correctness is not 198 affected. 200 1.4.2. NFSv4.1 Callback Operation 202 NFSv4.1 supports file delegation in a similar fashion to NFSv4.0, and 203 extends the repertoire of callback operations to manage pNFS layouts, 204 as discussed in Chapter 12 of [RFC5661]. 206 Further, NFSv4.1 requires that all transport connections are 207 initiated by NFSv4.1 clients. Thus, NFSv4.1 servers send callback 208 operations to clients in the backwards direction on connections that 209 NFSv4.1 clients establish with servers. 211 NFSv4.1 clients may establish separate transport connections for 212 forechannel and backchannel operation, or they may combine 213 forechannel and backchannel operation on the same transport 214 connection. 216 An NFSv4.1 client or server can signal its peer that a backchannel 217 capability is available on a given transport by sending a 218 CREATE_SESSION or BIND_CONN_TO_SESSION operation. When an RDMA 219 transport is used for the forechannel, an NFSv4.1 client must 220 additionally connect using a transport with bi-directional RPC 221 capability to use as a backchannel. Without a bi-directional RPC- 222 over-RDMA capability, TCP is the only choice at present for an 223 NFSv4.1 backchannel connection. 225 Some implementations prefer using a single combined transport. This 226 simplifies connection establishment and recovery during network 227 partitions or when one endpoint restarts. 229 Like NFSv4.0, if a backchannel is not in use, an NFSv4.1 server does 230 not grant delegations. But because of its reliance on callback 231 operations to manage pNFS layout state, pNFS operation is impossible 232 without a backchannel. 234 1.5. Design Considerations 235 1.5.1. Backwards Compatibility 237 Existing clients that implement RPC-over-RDMA version 1 should 238 interoperate correctly with servers that implement RPC-over-RDMA with 239 backwards direction support, and vice versa. 241 We prefer to avoid altering the RPC-over-RDMA version 1 XDR 242 specification. Keeping the XDR the same enables existing RPC-over- 243 RDMA version 1 implemenations to continue to interoperate with 244 implementations that support operation in the backwards direction. 246 1.5.2. Performance Impact 248 Support for operation in the backwards direction should never impact 249 the performance or scalability of forward direction operation, where 250 the bulk of ONC RPC transport activity typically occurs. 252 1.5.3. Client Complexity 254 RDMA transfers involve one endpoint exposing a portion of its memory 255 to the other, which then drives RDMA READ and WRITE operations to 256 access or modify the exposed memory. NFS clients expose their 257 memory, and NFS servers initiate RDMA data transfers. 259 We prefer to avoid having the server expose its memory to clients, 260 and to avoid introducing client complexity to drive RDMA operations. 262 1.5.4. Payload Size 264 Perhaps the only extant use case for backwards direction ONC RPC 265 messages is the NFSv4.1 backchannel. Our conventions can leverage 266 certain characteristics of NFSv4.1 callback operations. These 267 operations typically do not bear large argument or result payloads, 268 and are infrequent relative to forechannel operations. 270 Small RPC-over-RDMA messages are conveyed using only RDMA SEND, 271 without the complexity overhead of managing chunks. If only SEND is 272 permitted for backwards direction message, an RDMA_NOMSG type 273 message, which requires the use of a chunk, cannot be used to convey 274 a backwards direction message. 276 The price for this simplicity is that no backwards direction message 277 can be larger than the forward direction's receive buffer size 278 (typically 1024 bytes). 280 Stipulating such a limit on backwards direction message size assumes 281 either that Upper Layer Protocol consumers of backwards direction 282 messages can advertise this limit to peers; or that ULP consumers can 283 agree by convention on a maximum size of their backchannel payloads. 285 In addition, using only inline forms of RPC-over-RDMA messages and 286 never populating the RPC-over-RDMA chunk lists means that the RPC 287 header's msg_type field is always at a fixed location in messages 288 flowing in the backwards direction, allowing efficient detection of 289 the direction of an RPC-over-RDMA message. 291 With few exceptions, NFSv4.1 servers can break down callback requests 292 so they fit within this limit. There are a few potentially large 293 NFSv4.1 callback operations, such as a CB_GETATTR operation where a 294 large ACL must be conveyed. Although we are not aware of any NFSv4.1 295 implementation that uses CB_GETATTR, this state of affairs is not 296 guaranteed in perpetuity. 298 2. Conventions For Backwards Operation 300 Performing backwards direction ONC RPC operations over an RPC-over- 301 RDMA transport can be accomplished within limits by observing the 302 conventions described in the following subsections. For reference, 303 the XDR description of RPC-over-RDMA version 1 is contained in 304 Section 4.3 of [RFC5666]. 306 2.1. Flow Control 308 An RDMA SEND operation fails if the receiver has not pre-posted 309 enough buffers to receive the sent message. A sender might 310 retransmit the SEND operation, or it can choose to drop the 311 connection if message reception fails. 313 RPC-over-RDMA version 1 provides send flow control to prevent 314 overrunning the pre-posted receive buffers on a connection's receive 315 endpoint. This is fully discussed in Section 3.3 of [RFC5666]. 317 2.1.1. Forwards Credits 319 An RPC-over-RDMA credit is roughly the capability to handle one RPC- 320 over-RDMA call. Each forward direction RPC-over-RDMA call requests a 321 number of credits from the responder. Each forward direction reply 322 informs the caller how many credits the responder is prepared to 323 handle in total. The value of the request and grant are carried in 324 each RPC-over-RDMA message's rdma_credit field. 326 Practically speaking, the critical value is the value of the 327 rdma_credit field in RPC-over-RDMA replies. When an caller is 328 operating correctly, it sends no more outstanding requests at a time 329 than the responder's advertised forward direction credit value. 331 2.1.2. Backwards Credits 333 Credits work the same way in the backwards direction as they do in 334 the forward direction. However, forward direction credits and 335 backwards direction credits are accounted separately. 337 In other words, the forward direction credit value is the same 338 whether or not there are backward direction resources associated with 339 an RPC-over-RDMA transport connection. The backwards direction 340 credit value MAY be different than the forwards direction credit 341 value. 343 A backwards direction caller (an RPC-over-RDMA service endpoint) 344 requests credits from the responder (an RPC-over-RDMA client 345 endpoint). The responder reports how many credits it can grant. 346 This is the number of backwards direction calls the responder is 347 prepared to handle at once. 349 When an RPC-over-RDMA server endpoint is operating correctly, it 350 sends no more outstanding requests at a time than the client 351 endpoint's advertised backwards direction credit value. 353 If a sender transmits a backward direction message that exceeds the 354 receiver's backwards direction credit limit, the receiver MAY drop 355 the transport connection, or it MAY return an RPC-over-RDMA error to 356 the sender. The rdma_credit field in a backwards direction RPC-over- 357 RDMA message MUST NOT contain the value zero. 359 2.2. Managing Receive Buffers 361 A transport endpoint must pre-post receive buffers before it can 362 receive and process incoming RPC-over-RDMA messages. If a sender 363 transmits a message for a receiver which has no prepared receive 364 buffer, the receiver MUST drop the transport connection (?). This is 365 true no matter which direction a message flows. 367 2.2.1. Client Receive Buffers 369 Typically an RPC-over-RDMA caller posts only as many receive buffers 370 as there are outstanding RPC calls. A client endpoint without 371 backwards direction support might therefore at times have no pre- 372 posted receive buffers. 374 To receive incoming backwards direction calls, an RPC-over-RDMA 375 client endpoint must pre-post enough additional receive buffers to 376 match its backwards direction credit advertisement. 378 When an RDMA transport connection is lost, all active receive buffers 379 are flushed and are no longer available to receive incoming messages. 380 When a fresh transport connection is established, a client endpoint 381 must re-post a receive buffer to handle the reply for each 382 retransmitted forward direction call, and a full set of receive 383 buffers to handle backwards direction calls. 385 2.2.2. Server Receive Buffers 387 A forward direction RPC-over-RDMA service endpoint posts as many 388 receive buffers as it expects incoming forward direction calls. That 389 is, it posts no fewer buffers than the number of RPC-over-RDMA 390 credits it advertises in the rdma_credit field of forward direction 391 RPC replies. 393 To receive incoming backwards direction replies, an RPC-over-RDMA 394 server endpoint must pre-post a receive buffer for each backwards 395 direction call it sends. 397 When the existing transport connection is lost, all active receive 398 buffers are flushed and are no longer available to receive incoming 399 messages. When a fresh transport connection is established, a server 400 endpoint must re-post a receive buffer to handle the reply for each 401 retransmitted backwards direction call, and a full set of receive 402 buffers for receiving forward direction calls. 404 2.2.3. In the Absense of Backwards Direction Support 406 An RPC-over-RDMA transport endpoint might not support backwards 407 direction operation. There might be no mechanism in the 408 implementation to do so. Or the Upper Layer Protocol consumer might 409 not yet have configured the transport to handle backwards direction 410 traffic. 412 Since a receiver may drop the transport connection after receiving a 413 message it was not prepared for, a denial-of-service could result if 414 a sender continues to send backchannel messages after every transport 415 reconnect. 417 Generally, for RPC-over-RDMA version 1 transports, the Upper Layer 418 Protocol consumer is responsible for informing its peer when it has 419 no support for the backwards direction. Otherwise even a simple 420 backwards direction NULL probe from a peer results in a lost 421 connection. 423 For NFSv4.1, there is a built-in safety net for this case. An 424 NFSv4.1 server should never send backchannel messages to an NFSv4.1 425 client before the NFSv4.1 client has sent a CREATE_SESSION or a 426 BIND_CONN_TO_SESSION operation. As long as an NFSv4.1 client has 427 prepared appropriate backchannel resources before sending one of 428 these operations, denial-of-service is avoided. Legacy versions of 429 NFS should never send backchannel operations. 431 Therefore, an Upper Layer Protocol consumer MUST NOT perform 432 backwards direction ONC RPC operations unless the peer consumer has 433 signaled it is prepared to handle them. A description of the Upper 434 Layer Protocol mechanisms used for this signal is not in the scope of 435 this document. 437 2.3. Backwards Direction Message Size 439 RPC-over-RDMA backwards direction messages are transmitted and 440 received using the same buffers as messages in the forward direction. 441 Therefore they are constrained to be no larger than receive buffers 442 posted for forward messages. Typical implementations have chosen to 443 use 1024-byte buffers. 445 It is expected that the Upper Layer Protocol consumer establishes an 446 appropriate payload size limit, either by advertising that size limit 447 to its peers, or by convention. That way, backwards direction 448 messages do not exceed the size of receive buffers at either 449 endpoint. 451 If a sender transmits a backwards direction message that is larger 452 than the receiver is prepared for, or the message is too small to 453 convey a complete and valid RPC-over-RDMA and RPC message, the 454 receiver MUST drop the transport connection. 456 2.4. Sending A Backwards Direction Call 458 To form a backwards direction RPC-over-RDMA call message on an RPC- 459 over-RDMA version 1 transport, an ONC RPC service endpoint constructs 460 an RPC-over-RDMA header containing a fresh RPC XID in the rdma_xid 461 field. The rdma_vers field MUST contain the value one. The number 462 of requested credits is placed in the rdma_credit field (see 463 Section 2.1). 465 The rdma_proc field in the RPC-over-RDMA header MUST contain the 466 value RDMA_MSG. All three chunk lists MUST be empty. 468 The ONC RPC call header MUST follow immediately, starting with the 469 same XID value that is present in the RPC-over-RDMA header. The call 470 header's msg_type field MUST contain the value CALL. 472 2.5. Sending A Backwards Direction Reply 474 To form a backwards direction RPC-over-RDMA reply message on an RPC- 475 over-RDMA version 1 transport, an ONC RPC client endpoint constructs 476 an RPC-over-RDMA header containing a copy of the matching ONC RPC 477 call's RPC XID in the rdma_xid field. The rdma_vers field MUST 478 contain the value one. The number of granted credits is placed in 479 the rdma_credit field (see Section 2.1). 481 The rdma_proc field in the RPC-over-RDMA header MUST contain the 482 value RDMA_MSG. All three chunk lists MUST be empty. 484 The ONC RPC reply header MUST follow immediately, starting with the 485 same XID value that is present in the RPC-over-RDMA header. The 486 reply header's msg_type field MUST contain the value REPLY. 488 3. Limits To This Approach 490 3.1. Payload Size 492 The major drawback to the approach described in this document is the 493 limit on payload size in backwards direction requests. 495 o Some NFSv4.1 callback operations can have potentially large 496 arguments or results. For example, CB_GETATTR on a file with a 497 large ACL; or CB_NOTIFY, which can provide a large, complex 498 argument. 500 o Any backwards direction operation protected by RPCSEC_GSS may have 501 additional header information that makes it difficult to send 502 backwards direction operations with large arguments or results. 504 o Larger payloads could potentially require the use of RDMA data 505 transfers, which are complex and make it more difficult to detect 506 backwards direction requests. The msg_type field in the ONC RPC 507 header would no longer be at a fixed location in backwards 508 direction requests. 510 3.2. Preparedness To Handle Backwards Requests 512 A second drawback is the exposure of the client transport endpoint to 513 backwards direction calls before it has posted receive buffers to 514 handle them. 516 Clients that do not support backwards direction operation typically 517 drop messages they do not recognize. However, this does not allow 518 bi-direction-capable servers to quickly identify clients that cannot 519 handle backwards direction requests. 521 The conventions in this document rely on Upper Layer Protocol 522 consumers to decide when backwards direction transport operation is 523 appropriate. 525 3.3. Long Term 527 To address these limitations in the long run, we feel a revision of 528 the RPC-over-RDMA version 1 XDR is required, and that using 529 conventions to enable backwards direction operation is therefore a 530 transitional approach which is appropriate only while RPC-over-RDMA 531 version 1 is the predominantly deployed version of the RPC-over-RDMA 532 protocol. 534 4. Security Considerations 536 As a consequence of limiting the size of backwards direction RPC- 537 over-RDMA messages, the use of RPCSEC_GSS integrity and 538 confidentiality services (see [RFC2203]) may be challenging in the 539 backwards direction due to the size of the additional RPC header 540 information required for RPCSEC_GSS. 542 5. IANA Considerations 544 This document does not require actions by IANA. 546 6. Acknowledgements 548 The author of this document gratefully acknowledges the contributions 549 of Tom Talpey, Dai Ngo, Karen Deitke, Chunli Zhang, Dave Noveck, and 550 Bill Baker. 552 7. Normative References 554 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 555 Requirement Levels", BCP 14, RFC 2119, March 1997. 557 [RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol 558 Specification", RFC 2203, September 1997. 560 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 561 Specification Version 2", RFC 5531, May 2009. 563 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 564 System (NFS) Version 4 Minor Version 1 Protocol", RFC 565 5661, January 2010. 567 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 568 Transport for Remote Procedure Call", RFC 5666, January 569 2010. 571 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 572 Version 4 Protocol", RFC 7530, March 2015. 574 Author's Address 576 Charles Lever 577 Oracle Corporation 578 1015 Granger Avenue 579 Ann Arbor, MI 48104 580 US 582 Phone: +1 734 274 2396 583 Email: chuck.lever@oracle.com