idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-cm-pvt-data-08.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 21, 2020) is 1523 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC-TBD' is mentioned on line 454, but not defined -- Possible downref: Non-RFC (?) normative reference: ref. 'IBA' -- Obsolete informational reference (is this intentional?): RFC 5666 (Obsoleted by RFC 8166) Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Updates: 8166 (if approved) February 21, 2020 5 Intended status: Standards Track 6 Expires: August 24, 2020 8 RDMA Connection Manager Private Data For RPC-Over-RDMA Version 1 9 draft-ietf-nfsv4-rpcrdma-cm-pvt-data-08 11 Abstract 13 This document specifies the format of Remote Direct Memory Access - 14 Connection Manager (RDMA-CM) Private Data exchanged between RPC-over- 15 RDMA version 1 peers as part of establishing a connection. The 16 addition of the private data payload specified in this document is an 17 optional extension that does not alter the RPC-over-RDMA version 1 18 protocol. This document updates RFC 8166. 20 Status of This Memo 22 This Internet-Draft is submitted in full conformance with the 23 provisions of BCP 78 and BCP 79. 25 Internet-Drafts are working documents of the Internet Engineering 26 Task Force (IETF). Note that other groups may also distribute 27 working documents as Internet-Drafts. The list of current Internet- 28 Drafts is at https://datatracker.ietf.org/drafts/current/. 30 Internet-Drafts are draft documents valid for a maximum of six months 31 and may be updated, replaced, or obsoleted by other documents at any 32 time. It is inappropriate to use Internet-Drafts as reference 33 material or to cite them other than as "work in progress." 35 This Internet-Draft will expire on August 24, 2020. 37 Copyright Notice 39 Copyright (c) 2020 IETF Trust and the persons identified as the 40 document authors. All rights reserved. 42 This document is subject to BCP 78 and the IETF Trust's Legal 43 Provisions Relating to IETF Documents 44 (https://trustee.ietf.org/license-info) in effect on the date of 45 publication of this document. Please review these documents 46 carefully, as they describe your rights and restrictions with respect 47 to this document. Code Components extracted from this document must 48 include Simplified BSD License text as described in Section 4.e of 49 the Trust Legal Provisions and are provided without warranty as 50 described in the Simplified BSD License. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 55 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3 56 3. Advertised Transport Properties . . . . . . . . . . . . . . . 3 57 3.1. Inline Threshold Size . . . . . . . . . . . . . . . . . . 4 58 3.2. Remote Invalidation . . . . . . . . . . . . . . . . . . . 4 59 4. Private Data Message Format . . . . . . . . . . . . . . . . . 5 60 4.1. Using the R Field . . . . . . . . . . . . . . . . . . . . 7 61 4.2. Send and Receive Size Values . . . . . . . . . . . . . . 7 62 5. Interoperability Considerations . . . . . . . . . . . . . . . 7 63 5.1. Interoperability with RPC-over-RDMA Version 1 64 Implementations . . . . . . . . . . . . . . . . . . . . . 8 65 5.2. Interoperability Amongst RDMA Transports . . . . . . . . 8 66 6. Updating the Message Format . . . . . . . . . . . . . . . . . 8 67 7. Security Considerations . . . . . . . . . . . . . . . . . . . 9 68 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 69 8.1. Guidance for Designated Experts . . . . . . . . . . . . . 10 70 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 71 9.1. Normative References . . . . . . . . . . . . . . . . . . 11 72 9.2. Informative References . . . . . . . . . . . . . . . . . 12 73 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 12 74 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13 76 1. Introduction 78 The RPC-over-RDMA version 1 transport protocol [RFC8166] enables 79 payload data transfer using Remote Direct Memory Access (RDMA) for 80 upper-layer protocols based on Remote Procedure Calls (RPC) 81 [RFC5531]. The terms "Remote Direct Memory Access" (RDMA) and 82 "Direct Data Placement" (DDP) are introduced in [RFC5040]. 84 The two most immediate shortcomings of RPC-over-RDMA version 1 are: 86 o Setting up an RDMA data transfer (via RDMA Read or Write) can be 87 costly. The small default size of messages transmitted using RDMA 88 Send forces the use of RDMA Read or Write operations even for 89 relatively small messages and data payloads. 90 The original specification of RPC-over-RDMA version 1 provided an 91 out-of-band protocol for passing inline threshold values between 92 connected peers [RFC5666]. However, [RFC8166] eliminated support 93 for this protocol making it unavailable for this purpose. 95 o Unlike most other contemporary RDMA-enabled storage protocols, 96 there is no facility in RPC-over-RDMA version 1 that enables the 97 use of remote invalidation [RFC5042]. 99 Each RPC-over-RDMA version 1 transport header follows the External 100 Data Representation (XDR) [RFC4506] definition specified in 101 [RFC8166]. However, RPC-over-RDMA version 1 has no means of 102 extending this definition in such a way that interoperability with 103 existing implementations is preserved. As a result, an out-of-band 104 mechanism is needed to help relieve these constraints for existing 105 RPC-over-RDMA version 1 implementations. 107 This document specifies a simple, non-XDR-based message format 108 designed to be passed between RPC-over-RDMA version 1 peers at the 109 time each RDMA transport connection is first established. The 110 mechanism assumes that the underlying RDMA transport has a private 111 data field that is passed between peers at connection time, such as 112 is present in the iWARP protocol (described in Section 7.1 of 113 [RFC5044]) or the InfiniBand Connection Manager [IBA]. 115 To enable current RPC-over-RDMA version 1 implementations to 116 interoperate with implementations that support the private message 117 format described in this document, implementation of the private data 118 message is OPTIONAL. When the private data message has been 119 successfully exchanged, peers may choose to perform extended RDMA 120 semantics. However, the private message format does not alter the 121 XDR definition specified in [RFC8166]. 123 The message format is intended to be further extensible within the 124 normal scope of such IETF work (see Section 6 for further details). 125 Section 8 of this document defines an IANA registry for this purpose. 126 In addition, interoperation between implementations of RPC-over-RDMA 127 version 1 that present this message format to peers and those that do 128 not recognize this message format is guaranteed. 130 2. Requirements Language 132 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 133 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 134 "OPTIONAL" in this document are to be interpreted as described in BCP 135 14 [RFC2119] [RFC8174] when, and only when, they appear in all 136 capitals, as shown here. 138 3. Advertised Transport Properties 139 3.1. Inline Threshold Size 141 Section 3.3.2 of [RFC8166] defines the term "inline threshold." An 142 inline threshold is the maximum number of bytes that can be 143 transmitted using one RDMA Send and one RDMA Receive. There are a 144 pair of inline thresholds for a connection: a client-to-server 145 threshold and a server-to-client threshold. 147 If an incoming RDMA message exceeds the size of a receiver's inline 148 threshold, the receive operation fails and the RDMA provider 149 typically terminates the connection. To convey an RPC message larger 150 than the receiver's inline threshold without risking receive failure, 151 a sender must use explicit RDMA data transfer operations, which are 152 more expensive than an RDMA Send. See Sections 3.3 and 3.5 of 153 [RFC8166] for a complete discussion. 155 The default value of inline thresholds for RPC-over-RDMA version 1 156 connections is 1024 bytes (as defined in Section 3.3.3 of [RFC8166]). 157 This value is adequate for nearly all NFS version 3 procedures. 159 NFS version 4 COMPOUND operations [RFC7530] are larger on average 160 than NFS version 3 procedures [RFC1813], forcing clients to use 161 explicit RDMA operations for frequently-issued requests such as 162 LOOKUP and GETATTR. The use of RPCSEC_GSS security also increases 163 the average size of RPC messages, due to the larger size of 164 RPCSEC_GSS credential material included in RPC headers [RFC7861]. 166 If a sender and receiver could somehow agree on larger inline 167 thresholds, frequently-used RPC transactions avoid the cost of 168 explicit RDMA operations. 170 3.2. Remote Invalidation 172 After an RDMA data transfer operation completes, an RDMA consumer can 173 request that its peer's RDMA network interface card (RNIC) invalidate 174 the Steering Tag (STag) associated with the data transfer [RFC5042]. 176 An RDMA consumer requests remote invalidation by posting an RDMA Send 177 With Invalidate Work Request in place of an RDMA Send Work Request. 178 Each RDMA Send With Invalidate carries one STag to invalidate. The 179 receiver of an RDMA Send With Invalidate performs the requested 180 invalidation and then reports that invalidation as part of the 181 completion of a waiting Receive Work Request. 183 If both peers support remote invalidation, an RPC-over-RDMA responder 184 might use remote invalidation when replying to an RPC request that 185 provided chunks. Because one of the chunks has already been 186 invalidated, finalizing the results of the RPC is made simpler and 187 faster. 189 However, there are some important caveats which contraindicate the 190 blanket use of remote invalidation: 192 o Remote invalidation is not supported by all RNICs. 194 o Not all RPC-over-RDMA responder implementations can generate RDMA 195 Send With Invalidate Work Requests. 197 o Not all RPC-over-RDMA requester implementations can recognize when 198 remote invalidation has occurred. 200 o On one connection in different RPC-over-RDMA transactions, or in a 201 single RPC-over-RDMA transaction, an RPC-over-RDMA requester can 202 expose a mixture of STags that may be invalidated remotely and 203 some that must not be. No indication is provided at the RDMA 204 layer as to which is which. 206 A responder therefore must not employ remote invalidation unless it 207 is aware of support for it in its own RDMA stack, and on the 208 requester. And, without altering the XDR structure of RPC-over-RDMA 209 version 1 messages, it is not possible to support remote invalidation 210 with requesters that mix STags that may and must not be invalidated 211 remotely in a single RPC or on the same connection. 213 There are some NFS/RDMA client implementations whose STags are always 214 safe to invalidate remotely. For such clients, indicating to the 215 responder that remote invalidation is always safe can enable such 216 invalidation without the need for additional protocol elements to be 217 defined. 219 4. Private Data Message Format 221 With an InfiniBand lower layer, for example, RDMA connection setup 222 uses a Connection Manager when establishing a Reliable Connection 223 [IBA]. When an RPC-over-RDMA version 1 transport connection is 224 established, the client (which actively establishes connections) and 225 the server (which passively accepts connections) populate the CM 226 Private Data field exchanged as part of CM connection establishment. 228 The transport properties exchanged via this mechanism are fixed for 229 the life of the connection. Each new connection presents an 230 opportunity for a fresh exchange. An implementation of the extension 231 described in this document MUST be prepared for the settings to 232 change upon a reconnection. 234 For RPC-over-RDMA version 1, the CM Private Data field is formatted 235 as described in the following subsection. RPC clients and servers 236 use the same format. If the capacity of the Private Data field is 237 too small to contain this message format or the underlying RDMA 238 transport is not managed by a Connection Manager, the CM Private Data 239 field cannot be used on behalf of RPC-over-RDMA version 1. 241 The first 8 octets of the CM Private Data field is to be formatted as 242 follows: 244 0 1 2 3 245 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 246 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 247 | Format Identifier | 248 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 249 | Version | Reserved |R| Send Size | Receive Size | 250 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 252 Format Identifier: This field contains a fixed 32-bit value that 253 identifies the content of the Private Data field as an RPC-over- 254 RDMA version 1 CM Private Data message. In RPC-over-RDMA version 255 1 Private Data, the value of this field is always 0xf6ab0e18, in 256 network byte order. The use of this field is further expanded 257 upon in Section 5.2. 259 Version: This 8-bit field contains a message format version number. 260 The value "1" in this field indicates that exactly eight octets 261 are present, that they appear in the order described in this 262 section, and that each has the meaning defined in this section. 263 Further considerations about the use of this field are discussed 264 in Section 6. 266 Reserved: This 7-bit field is unused. Senders MUST set these bits 267 to zero and receivers MUST ignore their value. 269 R: This 1-bit field indicates that the sender supports remote 270 invalidation. The field is set and interpreted as described in 271 Section 4.1. 273 Send Size: This 8-bit field contains an encoded value corresponding 274 to the maximum number of bytes this peer is prepared to transmit 275 in a single RDMA Send on this connection. The value is encoded as 276 described in Section 4.2. 278 Receive Size: This 8-bit field contains an encoded value 279 corresponding to the maximum number of bytes this peer is prepared 280 to receive with a single RDMA Receive on this connection. The 281 value is encoded as described in Section 4.2. 283 4.1. Using the R Field 285 The R field indicates limited support for remote invalidate as 286 described in Section 3.2. When both connection peers have set this 287 bit flag in their CM Private Data, the responder MAY use RDMA Send 288 With Invalidate when transmitting RPC Replies. Each RDMA Send With 289 Invalidate MUST invalidate an STag associated only with the XID in 290 the rdma_xid field of the RPC-over-RDMA Transport Header it carries. 292 When either peer on a connection clears this flag, the responder MUST 293 use only RDMA Send when transmitting RPC Replies. 295 4.2. Send and Receive Size Values 297 Inline threshold sizes from 1024 to 262144 octets can be represented 298 in the Send Size and Receive Size fields. The inline threshold 299 values provide a pair of 1024-octet-aligned maximum message lengths 300 that guarantee Send and Receive operations do not fail due to length 301 errors. 303 The minimum inline threshold for RPC-over-RDMA version 1 is 1024 304 octets (see Section 3.3.3 of [RFC8166]). The values in the Send Size 305 and Receive Size fields represent the unsigned number of additional 306 kilo-octets of length beyond the first 1024 octets. Thus, a sender 307 computes the encoded value by dividing its actual buffer size, in 308 octets, by 1024 and subtracting one from the result. A receiver 309 decodes an incoming Size value by performing the inverse set of 310 operations: it adds one to the encoded value and then multiplies that 311 result by 1024. 313 The client uses the smaller of its own send size and the server's 314 reported receive size as the client-to-server inline threshold. The 315 server uses the smaller of its own send size and the clients's 316 reported receive size as the server-to-client inline threshold. 318 5. Interoperability Considerations 320 The extension described in this document is designed to allow RPC- 321 over-RDMA version implementations that use CM Private Data to 322 interoperate fully with RPC-over-RDMA version 1 implementations that 323 do not exchange this information. Implementations that use this 324 extension must also interoperate fully with RDMA implementations that 325 use CM Private Data for other purposes. Realizing these goals 326 requires that implementations of this extension follow the practices 327 described in the rest of this section. 329 5.1. Interoperability with RPC-over-RDMA Version 1 Implementations 331 When a peer does not receive a CM Private Data message which conforms 332 to Section 4, it needs to act as if the remote peer supports only the 333 default RPC-over-RDMA version 1 settings, as defined in [RFC8166]. 334 In other words, the peer MUST behave as if a Private Data message was 335 received in which bit 15 of the Flags field is zero, and both Size 336 fields contain the value zero. 338 5.2. Interoperability Amongst RDMA Transports 340 The Format Identifier field defined in Section 4 is provided to 341 enable implementations to distinguish RPC-over-RDMA version 1 Private 342 Data from private data inserted at other layers, such as the private 343 data inserted by the iWARP MPAv2 enhancement described in [RFC6581]. 345 As part of connection establishment, the received private data buffer 346 is searched for the Format Identifier word. The offset of the Format 347 Identifier is not restricted to any alignment. If the RPC-over-RDMA 348 version 1 CM Private Data Format Identifier is not present, an RPC- 349 over-RDMA version 1 receiver MUST behave as if no RPC-over-RDMA 350 version 1 CM Private Data has been provided. 352 Once the RPC-over-RDMA version 1 CM Private Data Format Identifier is 353 found, the receiver parses the subsequent octets as RPC-over-RDMA 354 version 1 CM Private Data. As additional assurance that the private 355 data content is valid RPC-over-RDMA version 1 CM Private Data, the 356 receiver should check that the format version number field contains a 357 valid and recognized version number and the size of the private data 358 does not overrun the length of the buffer. 360 6. Updating the Message Format 362 Although the message format described in this document provides the 363 ability for the client and server to exchange particular information 364 about the local RPC-over-RDMA implementation, it is possible that 365 there will be a future need to exchange additional properties. This 366 would make it necessary to extend or otherwise modify the format 367 described in this document. 369 Any modification faces the problem of interoperating properly with 370 implementations of RPC-over-RDMA version 1 that are unaware of the 371 existence of the new format. These include implementations that that 372 do not recognize the exchange of CM Private Data as well as those 373 that recognize only the format described in this document. 375 Given the message format described in this document, these 376 interoperability constraints could be met by the following sorts of 377 new message formats: 379 o A format which uses a different value for the first four bytes of 380 the format, as provided for in the registry described in 381 Section 8. 383 o A format which uses the same value for the Format Identifier field 384 and a value other than one (1) in the Version field. 386 Although it is possible to reorganize the last three of the eight 387 bytes in the existing format, extended formats are unlikely to do so. 388 New formats would take the form of extensions of the format described 389 in this document with added fields starting at byte eight of the 390 format or changes to the definition of bits in the Reserved field. 392 7. Security Considerations 394 The reader is directed to the Security Considerations section of 395 [RFC8166] for background and further discussion. 397 The RPC-over-RDMA version 1 protocol framework depends on the 398 semantics of the Reliable Connected (RC) queue pair (QP) type, as 399 defined in Section 9.7.7 of [IBA]. The integrity of CM Private Data 400 and the authenticity of its source are ensured by the exclusive use 401 of RC queue pairs. Any attempt to interfere with or hijack data in 402 transit on an RC connection results in the RDMA provider terminating 403 the connection. 405 The Security Considerations section of [RFC5042] refers the reader to 406 further relevant discussion of generic RDMA transport security. That 407 document recommends IPsec as the default transport layer security 408 solution. When deployed with iWARP, IPsec establishes a protected 409 channel before any iWARP operations are exchanged, thus it protects 410 the exchange of Private Data that occurs as each QP is established. 411 However, IPsec is not available for InfiniBand or RoCE deployments. 412 Those fabrics rely on physical security and cyclic redundancy checks 413 to protect network traffic. 415 Exchanging the information contained in the Private Message format 416 defined in this document does not expose upper-layer payloads to an 417 attacker. Furthermore, the behavior changes that occur as a result 418 of processing the CM Private Data format described in the current 419 document do not introduce any new risk of exposure of upper-layer 420 payload data. 422 Improperly setting one of the fields in a version 1 Private Message 423 can result in an increased risk of disconnection (i.e., self-imposed 424 Denial of Service). A similar risk can arise if non-RPC-over-RDMA CM 425 Private Data inadvertently contains the Format Identifier that 426 identifies this protocol's data structure. Additional checking of 427 incoming Private Data, as described in Section 5.2, can help reduce 428 this risk. 430 In addition to describing the structure of a new format version, any 431 document that extends the Private Data format described in the 432 current document must discuss security considerations of new data 433 items exchanged between connection peers. Such documents should also 434 explore the risks of erroneously identifying non-RPC-over-RDMA CM 435 Private Data as the new format. 437 8. IANA Considerations 439 In accordance with [RFC8126], the author requests that IANA create a 440 new registry in the "Remote Direct Data Placement" Protocol Category 441 Group. The new registry is to be called the "RDMA-CM Private Data 442 Identifier Registry". This is a registry of 32-bit numbers that 443 identify the upper-layer protocol associated with data that appears 444 in the application-specific RDMA-CM Private Data area. The fields in 445 this registry include: Format Identifier, Format Length (in octets), 446 Description, and Reference. 448 The initial contents of this registry are a single entry: 450 +---------------+--------+------------------------------+-----------+ 451 | Format | Length | Description | Reference | 452 | Identifier | | | | 453 +---------------+--------+------------------------------+-----------+ 454 | 0xf6ab0e18 | 8 | RPC-over-RDMA version 1 CM | [RFC-TBD] | 455 | | | Private Data | | 456 +---------------+--------+------------------------------+-----------+ 458 Table 1: RDMA-CM Private Data Identifier Registry 460 IANA is to assign subsequent new entries in this registry using the 461 Specification Required policy as defined in Section 4.6 of [RFC8126]. 463 8.1. Guidance for Designated Experts 465 The Designated Expert (DE), appointed by the IESG, should ascertain 466 the existence of suitable documentation that defines the semantics 467 and format of the private data, and verify that the document is 468 permanently and publicly available. Documentation produced outside 469 the IETF must not conflict with work that is active or already 470 published within the IETF. The new Reference field should contain a 471 reference to that documentation. 473 The Description field should contain the name of the upper-layer 474 protocol that generates and uses the private data. 476 The DE should assign a new Format Identifier so that it does not 477 conflict with existing entries in this registry, and so that it is 478 not likely to be mistaken as part of the payload of other registered 479 formats. 481 The DE shall post the request to the nfsv4 WG mailing list (or a 482 successor to that list, if such a list exists), for comment and 483 review. The DE shall approve or deny the request and publish notice 484 of the decision within 30 days. 486 9. References 488 9.1. Normative References 490 [IBA] InfiniBand Trade Association, "InfiniBand Architecture 491 Specification Volume 1", Release 1.3, March 2015. 493 Available from https://www.infinibandta.org/ 495 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 496 Requirement Levels", BCP 14, RFC 2119, 497 DOI 10.17487/RFC2119, March 1997, 498 . 500 [RFC4506] Eisler, M., Ed., "XDR: External Data Representation 501 Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 502 2006, . 504 [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. 505 Garcia, "A Remote Direct Memory Access Protocol 506 Specification", RFC 5040, DOI 10.17487/RFC5040, October 507 2007, . 509 [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement 510 Protocol (DDP) / Remote Direct Memory Access Protocol 511 (RDMAP) Security", RFC 5042, DOI 10.17487/RFC5042, October 512 2007, . 514 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 515 Writing an IANA Considerations Section in RFCs", BCP 26, 516 RFC 8126, DOI 10.17487/RFC8126, June 2017, 517 . 519 [RFC8166] Lever, C., Ed., Simpson, W., and T. Talpey, "Remote Direct 520 Memory Access Transport for Remote Procedure Call Version 521 1", RFC 8166, DOI 10.17487/RFC8166, June 2017, 522 . 524 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 525 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 526 May 2017, . 528 9.2. Informative References 530 [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS 531 Version 3 Protocol Specification", RFC 1813, 532 DOI 10.17487/RFC1813, June 1995, 533 . 535 [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. 536 Carrier, "Marker PDU Aligned Framing for TCP 537 Specification", RFC 5044, DOI 10.17487/RFC5044, October 538 2007, . 540 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 541 Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, 542 May 2009, . 544 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 545 Transport for Remote Procedure Call", RFC 5666, 546 DOI 10.17487/RFC5666, January 2010, 547 . 549 [RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S. 550 Wise, "Enhanced Remote Direct Memory Access (RDMA) 551 Connection Establishment", RFC 6581, DOI 10.17487/RFC6581, 552 April 2012, . 554 [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System 555 (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, 556 March 2015, . 558 [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) 559 Security Version 3", RFC 7861, DOI 10.17487/RFC7861, 560 November 2016, . 562 Acknowledgments 564 Thanks to Christoph Hellwig and Devesh Sharma for suggesting this 565 approach, and to Tom Talpey and Dave Noveck for their expert comments 566 and review. The author also wishes to thank Bill Baker and Greg 567 Marsden for their support of this work. Also, thanks to expert 568 reviewers Sean Hefty and Dave Minturn. 570 Special thanks go to document shepherd Brian Pawlowski, Transport 571 Area Director Magnus Westerlund, NFSV4 Working Group Chairs David 572 Noveck and Spencer Shepler, and NFSV4 Working Group Secretary Thomas 573 Haynes. 575 Author's Address 577 Charles Lever 578 Oracle Corporation 579 United States of America 581 Email: chuck.lever@oracle.com