idnits 2.17.1 draft-cel-nfsv4-rpcrdma-bidirection-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 20, 2015) is 3263 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 5661 (Obsoleted by RFC 8881) ** Obsolete normative reference: RFC 5666 (Obsoleted by RFC 8166) Summary: 2 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 NFSv4 C. Lever 3 Internet-Draft Oracle 4 Intended status: Experimental May 20, 2015 5 Expires: November 21, 2015 7 Size-Limited Bi-directional Remote Procedure Call On Remote Direct 8 Memory Access Transports 9 draft-cel-nfsv4-rpcrdma-bidirection-01 11 Abstract 13 Recent minor versions of NFSv4 work best when ONC RPC transports can 14 send ONC RPC transactions in both directions. This document 15 describes conventions that enable RPC-over-RDMA version 1 transport 16 endpoints to interoperate when operation in both directions is 17 necessary. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at http://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on November 21, 2015. 36 Copyright Notice 38 Copyright (c) 2015 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (http://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 55 1.2. Scope Of This Document . . . . . . . . . . . . . . . . . 3 56 1.3. Understanding RPC Direction . . . . . . . . . . . . . . . 3 57 1.3.1. Forward Direction . . . . . . . . . . . . . . . . . . 4 58 1.3.2. Backward Direction . . . . . . . . . . . . . . . . . 4 59 1.3.3. Bi-direction . . . . . . . . . . . . . . . . . . . . 4 60 1.3.4. XID Values . . . . . . . . . . . . . . . . . . . . . 4 61 1.4. Rationale For RPC-over-RDMA Bi-Direction . . . . . . . . 5 62 1.4.1. NFSv4.0 Callback Operation . . . . . . . . . . . . . 5 63 1.4.2. NFSv4.1 Callback Operation . . . . . . . . . . . . . 6 64 1.5. Design Considerations . . . . . . . . . . . . . . . . . . 6 65 1.5.1. Backward Compatibility . . . . . . . . . . . . . . . 7 66 1.5.2. Performance Impact . . . . . . . . . . . . . . . . . 7 67 1.5.3. Server Memory Security . . . . . . . . . . . . . . . 7 68 1.5.4. Payload Size . . . . . . . . . . . . . . . . . . . . 7 69 2. Conventions For Backward Operation . . . . . . . . . . . . . 8 70 2.1. Flow Control . . . . . . . . . . . . . . . . . . . . . . 8 71 2.1.1. Forward Credits . . . . . . . . . . . . . . . . . . . 8 72 2.1.2. Backward Credits . . . . . . . . . . . . . . . . . . 9 73 2.2. Managing Receive Buffers . . . . . . . . . . . . . . . . 9 74 2.2.1. Client Receive Buffers . . . . . . . . . . . . . . . 9 75 2.2.2. Server Receive Buffers . . . . . . . . . . . . . . . 10 76 2.2.3. In the Absense of Backward Direction Support . . . . 10 77 2.3. Backward Direction Message Size . . . . . . . . . . . . . 11 78 2.4. Sending A Backward Direction Call . . . . . . . . . . . . 11 79 2.5. Sending A Backward Direction Reply . . . . . . . . . . . 12 80 3. Limits To This Approach . . . . . . . . . . . . . . . . . . . 12 81 3.1. Payload Size . . . . . . . . . . . . . . . . . . . . . . 12 82 3.2. Preparedness To Handle Backward Requests . . . . . . . . 13 83 3.3. Long Term . . . . . . . . . . . . . . . . . . . . . . . . 13 84 4. Security Considerations . . . . . . . . . . . . . . . . . . . 13 85 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 86 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 13 87 7. Normative References . . . . . . . . . . . . . . . . . . . . 13 88 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 14 90 1. Introduction 91 1.1. Requirements Language 93 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 94 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 95 document are to be interpreted as described in [RFC2119]. 97 1.2. Scope Of This Document 99 This document describes a set of experimental conventions that apply 100 to RPC-over-RDMA version 1, specified in [RFC5666]. When observed, 101 these conventions enable RPC-over-RDMA version 1 endpoints to 102 concurrently handle RPC transactions that flow from client to server 103 from and server to client. 105 No changes to the RPC-over-RDMA version 1 protocol definition are 106 needed. Therefore this document does not update [RFC5666]. 108 The purpose of this document is to permit interoperable prototype 109 implementations of bi-directional RPC-over-RDMA, enabling NFSv4.1 110 (and later NFS minor versions) on RDMA transports. 112 Providing an Upper Layer Binding for NFSv4.x callback operations is 113 not in the scope of this document. 115 1.3. Understanding RPC Direction 117 The ONC RPC protocol, as described in [RFC5531], is fundamentally a 118 message-passing protocol involving one server and perhaps multiple 119 clients. ONC RPC transactions are made up of two types of messages. 121 A CALL message, or "call", requests work. A call is designated by 122 the value CALL in the message's msg_type field. An arbitrary unique 123 value is placed in the message's xid field. A host that originates a 124 call is referred to in this document as a "caller." 126 A REPLY message, or "reply", reports the results of work requested by 127 a call. A reply is designated by the value REPLY in the message's 128 msg_type field. The value contained in the message's xid field is 129 copied from the call whose results are being reported. A host that 130 emits a reply is referred to as a "responder." 132 RPC-over-RDMA is a connection-oriented RPC transport. When a 133 connection-oriented transport is used, ONC RPC client endpoints are 134 responsible for initiating transport connections, while ONC RPC 135 service endpoints wait passively for incoming connection requests. 136 We do not consider RPC direction on connectionless RPC transports in 137 this document. 139 1.3.1. Forward Direction 141 A traditional ONC RPC client is always a caller. A traditional ONC 142 RPC service is always a responder. This traditional form of ONC RPC 143 message passing is referred to as operation in the "forward 144 direction." 146 During forward direction operation, the ONC RPC client is responsible 147 for establishing transport connections. 149 1.3.2. Backward Direction 151 The ONC RPC standard does not forbid passing messages in the other 152 direction. An ONC RPC service endpoint can act as a caller, in which 153 case an ONC RPC client endpoint acts as a responder. This form of 154 message passing is referred to as operation in the "backward 155 direction." 157 During backward direction operation, the ONC RPC client is 158 responsible for establishing transport connections, even though ONC 159 RPC calls come from the ONC RPC server. 161 Notably, traditional ONC RPC clients and services are usually not 162 prepared for backward operation. ONC RPC clients and services are 163 heavily optimized to perform and scale well while handling traffic in 164 the forward direction. Not until recently has there been any need to 165 handle operation in the backward direction. 167 1.3.3. Bi-direction 169 A pair of endpoints may choose to use only forward or only backward 170 direction operations on a particular transport. Or, the endpoints 171 may send operations in both directions concurrently on the same 172 transport. 174 Bi-directional operation occurs when both transport endpoints act as 175 a caller and a responder at the same time. As above, the ONC RPC 176 client is responsible for establishing transport connections. 178 1.3.4. XID Values 180 Section 9 of [RFC5531] introduces the ONC RPC transaction identifier, 181 or "xid" for short. The value of an xid is interpreted in the 182 context of the message's msg_type field. 184 o The xid of a call is arbitrary but is unique among outstanding 185 calls from that caller. 187 o The xid of a reply always matches that of the initiating call. 189 A caller matches the xid value in each reply with a call it 190 previously sent. 192 1.3.4.1. XIDs with Bi-direction 194 During bi-directional operation, the forward and backward directions 195 use independent xid spaces. 197 In other words, a forward direction caller MAY use the same xid value 198 at the same time as a backward direction caller that occupies the 199 same transport connection. Though such concurrent requests use the 200 same xid value, they represent entirely unique ONC RPC transactions. 202 1.4. Rationale For RPC-over-RDMA Bi-Direction 204 1.4.1. NFSv4.0 Callback Operation 206 An NFSv4.0 client employs a traditional ONC RPC client to send NFS 207 requests to an NFSv4.0 server's traditional ONC RPC service 208 [RFC7530]. NFSv4.0 requests flow in the forward direction on a 209 connection established by the client. This connection is referred to 210 as a "forechannel." 212 NFSv4.0 introduces the use of callback operations, or "callbacks" for 213 short, in Section 10.2 of [RFC7530], for managing file delegation. 214 An NFSv4.0 server sets up a traditional ONC RPC client and an NFSv4.0 215 client sets up a traditional ONC RPC service to handle callbacks. 216 Callbacks flow in the forward direction on a connection established 217 by the server. This connection is distinct from connections being 218 used as forechannels. This connection is referred to as a 219 "backchannel." 221 When an RDMA transport is used as a forechannel, an NFSv4.0 client 222 typically provides a TCP callback service. The client's SETCLIENTID 223 operation advertises the callback service endpoint with a "tcp" or 224 "tcp6" netid. The server then connects to this service using a TCP 225 socket. 227 NFSv4.0 is fully functional without a backchannel in place. The 228 server simply does not grant file delegations. There might be a 229 negative performance effect, but operational correctness is not 230 affected. 232 1.4.2. NFSv4.1 Callback Operation 234 NFSv4.1 supports file delegation in a similar fashion to NFSv4.0, and 235 extends the repertoire of callbacks to manage pNFS layouts, as 236 discussed in Chapter 12 of [RFC5661]. 238 For various reasons, NFSv4.1 requires that all transport connections 239 be initiated by NFSv4.1 clients. Therefore, NFSv4.1 servers send 240 callbacks to clients in the backward direction on connections 241 established by NFSv4.1 clients. 243 An NFSv4.1 client or server indicates to its peer that a backchannel 244 capability is available on a given transport when sending a 245 CREATE_SESSION or BIND_CONN_TO_SESSION operation. 247 NFSv4.1 clients may establish distinct transport connections for 248 forechannel and backchannel operation, or they may combine 249 forechannel and backchannel operation on one transport connection 250 using bi-directional operation. 252 When an RDMA transport is used as a forechannel, an NFSv4.1 client 253 must additionally connect using a transport with backward direction 254 capability to use as a backchannel. Without a backward direction 255 RPC-over-RDMA capability, TCP is the only choice at present for an 256 NFSv4.1 backchannel connection. 258 Some implementations prefer using a single combined transport (ie. a 259 transport that is capable of bi-directional operation). This 260 simplifies connection establishment and recovery during network 261 partitions or when one endpoint restarts. 263 As with NFSv4.0, if a backchannel is not in use, an NFSv4.1 server 264 does not grant delegations. But because of its reliance on callbacks 265 to manage pNFS layout state, pNFS operation is impossible without a 266 backchannel. 268 1.5. Design Considerations 270 As of this writing, the only use case for backward direction ONC RPC 271 messages is the NFSv4.1 backchannel. The conventions described in 272 this document take advantage of certain characteristics of NFSv4.1 273 callbacks. 275 NFSv4.1 callbacks typically do not bear large argument or result 276 payloads, or payloads that are sensitive to alignment. Callbacks are 277 infrequent relative to forechannel operations. 279 1.5.1. Backward Compatibility 281 Existing clients that implement RPC-over-RDMA version 1 should 282 interoperate correctly with servers that implement RPC-over-RDMA with 283 backward direction support, and vice versa. 285 We wish to avoid altering the RPC-over-RDMA version 1 XDR 286 specification. Keeping the XDR the same enables existing RPC-over- 287 RDMA version 1 implemenations to interoperate with implementations 288 that support operation in the backward direction. 290 1.5.2. Performance Impact 292 Support for operation in the backward direction should never impact 293 the performance or scalability of forward direction operation, where 294 the bulk of ONC RPC transport activity typically occurs. 296 1.5.3. Server Memory Security 298 RDMA transfers involve one endpoint exposing a portion of its memory 299 to the other endpoint, which then drives RDMA READ and WRITE 300 operations to access or modify the exposed memory. RPC-over-RDMA 301 client endpoints expose their memory, and RPC-over-RDMA server 302 endpoints initiate RDMA data transfer operations. 304 By avoiding RDMA transfers for backward direction operations, servers 305 do not expose their memory to clients. Further, there is no need to 306 introduce client complexity to drive RDMA transfers. 308 1.5.4. Payload Size 310 Small RPC-over-RDMA messages are conveyed using only RDMA SEND 311 operations. SEND is used to transmit both ONC RPC calls and replies. 313 To send a large payload, an RPC-over-RDMA client endpoint registers a 314 region of memory known as a chunk and transmits its coordinates to a 315 server endpoint, who uses an RDMA transfer to move data to or from 316 the client. See Sections 3.1, 3.2, and 3.4 of [RFC5666]. 318 To transmit RPC-over-RDMA messages larger than the receive buffer 319 size (typically 1024 bytes), a chunk must be used. For example, in 320 an RDMA_NOMSG type message, the entire RPC header and Upper Layer 321 payload are contained in chunks. See Section 5.1 of [RFC5666] for 322 details. 324 If chunks are not allowed to be used for conveying backward direction 325 messages, an RDMA_NOMSG type message cannot be used to convey a 326 backward direction message using the conventions described in this 327 document. Therefore, backward direction messages sent using the 328 conventions in this document can be no larger than a receive buffer. 330 Stipulating such a limit on backward direction message size assumes 331 that either Upper Layer Protocol consumers of backward direction 332 messages can advertise this limit to peers, or that ULP consumers can 333 agree by convention on a maximum size of their backchannel payloads. 335 In addition, using only inline forms of RPC-over-RDMA messages and 336 never populating the RPC-over-RDMA chunk lists means that the RPC 337 header's msg_type field is always at a fixed location in messages 338 flowing in the backward direction, allowing efficient detection of 339 the direction of an RPC-over-RDMA message. 341 With few exceptions, NFSv4.1 servers can break down callback requests 342 so they fit within this limit. There are a few potentially large 343 NFSv4.1 callback operations, such as a CB_GETATTR operation where a 344 large ACL must be conveyed. Although we are not aware of any NFSv4.1 345 implementation that uses CB_GETATTR, this state of affairs is not 346 guaranteed in perpetuity. 348 2. Conventions For Backward Operation 350 Performing backward direction ONC RPC operations over an RPC-over- 351 RDMA transport can be accomplished within limits by observing the 352 conventions described in the following subsections. For reference, 353 the XDR description of RPC-over-RDMA version 1 is contained in 354 Section 4.3 of [RFC5666]. 356 2.1. Flow Control 358 An RDMA SEND operation fails if the receiver has not pre-posted 359 enough buffers to receive the sent message. A sender might 360 retransmit the SEND operation, or it can choose to drop the 361 connection if message reception fails. 363 RPC-over-RDMA version 1 provides send flow control to prevent 364 overrunning the pre-posted receive buffers on a connection's receive 365 endpoint. This is fully discussed in Section 3.3 of [RFC5666]. 367 2.1.1. Forward Credits 369 An RPC-over-RDMA credit is roughly the capability to handle one RPC- 370 over-RDMA transaction. Each forward direction RPC-over-RDMA call 371 requests a number of credits from the responder. Each forward 372 direction reply informs the caller how many credits the responder is 373 prepared to handle in total. The value of the request and grant are 374 carried in each RPC-over-RDMA message's rdma_credit field. 376 Practically speaking, the critical value is the value of the 377 rdma_credit field in RPC-over-RDMA replies. When a caller is 378 operating correctly, it sends no more outstanding requests at a time 379 than the responder's advertised forward direction credit value. 381 2.1.2. Backward Credits 383 Credits work the same way in the backward direction as they do in the 384 forward direction. However, forward direction credits and backward 385 direction credits are accounted separately. 387 In other words, the forward direction credit value is the same 388 whether or not there are backward direction resources associated with 389 an RPC-over-RDMA transport connection. The backward direction credit 390 value MAY be different than the forward direction credit value. 392 A backward direction caller (an RPC-over-RDMA service endpoint) 393 requests credits from the responder (an RPC-over-RDMA client 394 endpoint). The responder reports how many credits it can grant. 395 This is the number of backward direction calls the responder is 396 prepared to handle at once. 398 When an RPC-over-RDMA server endpoint is operating correctly, it 399 sends no more outstanding requests at a time than the client 400 endpoint's advertised backward direction credit value. 402 If a sender transmits a backward direction message that exceeds the 403 receiver's backward direction credit limit, the receiver MAY drop the 404 transport connection, or it MAY return an RPC-over-RDMA error to the 405 sender. The rdma_credit field in a backward direction RPC-over-RDMA 406 message MUST NOT contain the value zero. 408 2.2. Managing Receive Buffers 410 An RPC-over-RDMA transport endpoint must pre-post receive buffers 411 before it can receive and process incoming RPC-over-RDMA messages. 412 If a sender transmits a message for a receiver which has no prepared 413 receive buffer, the receiver MUST drop the transport connection (?). 414 This is true no matter which direction a message flows. 416 2.2.1. Client Receive Buffers 418 Typically an RPC-over-RDMA caller posts only as many receive buffers 419 as there are outstanding RPC calls. A client endpoint without 420 backward direction support might therefore at times have no pre- 421 posted receive buffers. 423 To receive incoming backward direction calls, an RPC-over-RDMA client 424 endpoint must pre-post enough additional receive buffers to match its 425 backward direction credit advertisement. 427 When an RDMA transport connection is lost, all active receive buffers 428 are flushed and are no longer available to receive incoming messages. 429 When a fresh transport connection is established, a client endpoint 430 must re-post a receive buffer to handle the reply for each 431 retransmitted forward direction call, and a full set of receive 432 buffers to handle backward direction calls. 434 2.2.2. Server Receive Buffers 436 A forward direction RPC-over-RDMA service endpoint posts as many 437 receive buffers as it expects incoming forward direction calls. That 438 is, it posts no fewer buffers than the number of RPC-over-RDMA 439 credits it advertises in the rdma_credit field of forward direction 440 RPC replies. 442 To receive incoming backward direction replies, an RPC-over-RDMA 443 server endpoint must pre-post a receive buffer for each backward 444 direction call it sends. 446 When the existing transport connection is lost, all active receive 447 buffers are flushed and are no longer available to receive incoming 448 messages. When a fresh transport connection is established, a server 449 endpoint must re-post a receive buffer to handle the reply for each 450 retransmitted backward direction call, and a full set of receive 451 buffers for receiving forward direction calls. 453 2.2.3. In the Absense of Backward Direction Support 455 An RPC-over-RDMA transport endpoint might not support backward 456 direction operation. There might be no mechanism in the 457 implementation to do so. Or the Upper Layer Protocol consumer might 458 not yet have configured the transport to handle backward direction 459 traffic. 461 A receiver may drop the transport connection after receiving a 462 message it was not prepared for. Thus a denial-of-service could 463 result if a sender continues to send backchannel messages after every 464 transport reconnect to an endpoint that is not prepared to receive 465 them. 467 Generally, for RPC-over-RDMA version 1 transports, the Upper Layer 468 Protocol consumer is responsible for informing its peer when it has 469 no support for the backward direction. Otherwise even a simple 470 backward direction NULL probe from a peer results in a lost 471 connection. 473 An NFSv4.1 server should never send backchannel messages to an 474 NFSv4.1 client before the NFSv4.1 client has sent a CREATE_SESSION or 475 a BIND_CONN_TO_SESSION operation. As long as an NFSv4.1 client has 476 prepared appropriate backchannel resources before sending one of 477 these operations, denial-of-service is avoided. Legacy versions of 478 NFS should never send backchannel operations. 480 Therefore, an Upper Layer Protocol consumer MUST NOT perform backward 481 direction ONC RPC operations unless the peer consumer has indicated 482 it is prepared to handle them. A description of Upper Layer Protocol 483 mechanisms used for this indication is not in the scope of this 484 document. 486 2.3. Backward Direction Message Size 488 RPC-over-RDMA backward direction messages are transmitted and 489 received using the same buffers as messages in the forward direction. 490 Therefore they are constrained to be no larger than receive buffers 491 posted for forward messages. Typical implementations have chosen to 492 use 1024-byte buffers. 494 It is expected that the Upper Layer Protocol consumer establishes an 495 appropriate payload size limit, either by advertising that size limit 496 to its peers, or by convention. If that is done, backward direction 497 messages would not exceed the size of receive buffers at either 498 endpoint. 500 If a sender transmits a backward direction message that is larger 501 than the receiver is prepared for, or the message is too small to 502 convey a complete and valid RPC-over-RDMA and RPC message, the 503 receiver MUST drop the transport connection. 505 2.4. Sending A Backward Direction Call 507 To form a backward direction RPC-over-RDMA call message on an RPC- 508 over-RDMA version 1 transport, an ONC RPC service endpoint constructs 509 an RPC-over-RDMA header containing a fresh RPC XID in the rdma_xid 510 field (see Section 1.3.4 for full requirements). 512 The rdma_vers field MUST contain the value one. The number of 513 requested credits is placed in the rdma_credit field (see 514 Section 2.1). 516 The rdma_proc field in the RPC-over-RDMA header MUST contain the 517 value RDMA_MSG. All three chunk lists MUST be empty. 519 The ONC RPC call header MUST follow immediately, starting with the 520 same XID value that is present in the RPC-over-RDMA header. The call 521 header's msg_type field MUST contain the value CALL. 523 2.5. Sending A Backward Direction Reply 525 To form a backward direction RPC-over-RDMA reply message on an RPC- 526 over-RDMA version 1 transport, an ONC RPC client endpoint constructs 527 an RPC-over-RDMA header containing a copy of the matching ONC RPC 528 call's RPC XID in the rdma_xid field (see Section 1.3.4 for full 529 requirements). 531 The rdma_vers field MUST contain the value one. The number of 532 granted credits is placed in the rdma_credit field (see Section 2.1). 534 The rdma_proc field in the RPC-over-RDMA header MUST contain the 535 value RDMA_MSG. All three chunk lists MUST be empty. 537 The ONC RPC reply header MUST follow immediately, starting with the 538 same XID value that is present in the RPC-over-RDMA header. The 539 reply header's msg_type field MUST contain the value REPLY. 541 3. Limits To This Approach 543 3.1. Payload Size 545 The major drawback to the approach described in this document is the 546 limit on payload size in backward direction requests. 548 o Some NFSv4.1 callback operations can have potentially large 549 arguments or results. For example, CB_GETATTR on a file with a 550 large ACL; or CB_NOTIFY, which can provide a large, complex 551 argument. 553 o Any backward direction operation protected by RPCSEC_GSS may have 554 additional header information that makes it difficult to send 555 backward direction operations with large arguments or results. 557 o Larger payloads could potentially require the use of RDMA data 558 transfers, which are complex and make it more difficult to detect 559 backward direction requests. The msg_type field in the ONC RPC 560 header would no longer be at a fixed location in backward 561 direction requests. 563 3.2. Preparedness To Handle Backward Requests 565 A second drawback is the exposure of the client transport endpoint to 566 backward direction calls before it has posted receive buffers to 567 handle them. 569 Clients that do not support backward direction operation typically 570 drop messages they do not recognize. However, this does not allow 571 bi-direction-capable servers to quickly identify clients that cannot 572 handle backward direction requests. 574 The conventions in this document rely on Upper Layer Protocol 575 consumers to decide when backward direction transport operation is 576 appropriate. 578 3.3. Long Term 580 To address these limitations in the long run, we feel a revision of 581 the RPC-over-RDMA version 1 XDR is required, and that using 582 conventions to enable backward direction operation is therefore a 583 transitional approach which is appropriate only while RPC-over-RDMA 584 version 1 is the predominantly deployed version of the RPC-over-RDMA 585 protocol. 587 4. Security Considerations 589 As a consequence of limiting the size of backward direction RPC-over- 590 RDMA messages, the use of RPCSEC_GSS integrity and confidentiality 591 services (see [RFC2203]) in the backward direction may be challenging 592 due to the size of the additional RPC header information required for 593 RPCSEC_GSS. 595 5. IANA Considerations 597 This document does not require actions by IANA. 599 6. Acknowledgements 601 The author of this document gratefully acknowledges the contributions 602 of Tom Talpey, Dai Ngo, Karen Deitke, Chunli Zhang, Dave Noveck, and 603 Bill Baker. 605 7. Normative References 607 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 608 Requirement Levels", BCP 14, RFC 2119, March 1997. 610 [RFC2203] Eisler, M., Chiu, A., and L. Ling, "RPCSEC_GSS Protocol 611 Specification", RFC 2203, September 1997. 613 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 614 Specification Version 2", RFC 5531, May 2009. 616 [RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File 617 System (NFS) Version 4 Minor Version 1 Protocol", RFC 618 5661, January 2010. 620 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 621 Transport for Remote Procedure Call", RFC 5666, January 622 2010. 624 [RFC7530] Haynes, T. and D. Noveck, "Network File System (NFS) 625 Version 4 Protocol", RFC 7530, March 2015. 627 Author's Address 629 Charles Lever 630 Oracle Corporation 631 1015 Granger Avenue 632 Ann Arbor, MI 48104 633 US 635 Phone: +1 734 274 2396 636 Email: chuck.lever@oracle.com