idnits 2.17.1 draft-ietf-nfsv4-rpcrdma-cm-pvt-data-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (January 31, 2020) is 1519 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC-TBD' is mentioned on line 434, but not defined -- Obsolete informational reference (is this intentional?): RFC 5666 (Obsoleted by RFC 8166) Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever 3 Internet-Draft Oracle 4 Updates: 8166 (if approved) January 31, 2020 5 Intended status: Standards Track 6 Expires: August 3, 2020 8 RDMA Connection Manager Private Data For RPC-Over-RDMA Version 1 9 draft-ietf-nfsv4-rpcrdma-cm-pvt-data-07 11 Abstract 13 This document specifies the format of RDMA-CM Private Data exchanged 14 between RPC-over-RDMA version 1 peers as part of establishing a 15 connection. The addition of the private data payload specified in 16 this document is an optional extension that does not alter the RPC- 17 over-RDMA version 1 protocol. This document updates RFC 8166. 19 Status of This Memo 21 This Internet-Draft is submitted in full conformance with the 22 provisions of BCP 78 and BCP 79. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF). Note that other groups may also distribute 26 working documents as Internet-Drafts. The list of current Internet- 27 Drafts is at https://datatracker.ietf.org/drafts/current/. 29 Internet-Drafts are draft documents valid for a maximum of six months 30 and may be updated, replaced, or obsoleted by other documents at any 31 time. It is inappropriate to use Internet-Drafts as reference 32 material or to cite them other than as "work in progress." 34 This Internet-Draft will expire on August 3, 2020. 36 Copyright Notice 38 Copyright (c) 2020 IETF Trust and the persons identified as the 39 document authors. All rights reserved. 41 This document is subject to BCP 78 and the IETF Trust's Legal 42 Provisions Relating to IETF Documents 43 (https://trustee.ietf.org/license-info) in effect on the date of 44 publication of this document. Please review these documents 45 carefully, as they describe your rights and restrictions with respect 46 to this document. Code Components extracted from this document must 47 include Simplified BSD License text as described in Section 4.e of 48 the Trust Legal Provisions and are provided without warranty as 49 described in the Simplified BSD License. 51 Table of Contents 53 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 54 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3 55 3. Advertised Transport Properties . . . . . . . . . . . . . . . 3 56 3.1. Inline Threshold Size . . . . . . . . . . . . . . . . . . 3 57 3.2. Remote Invalidation . . . . . . . . . . . . . . . . . . . 4 58 4. Private Data Message Format . . . . . . . . . . . . . . . . . 5 59 4.1. Interoperability Considerations . . . . . . . . . . . . . 6 60 4.1.1. Interoperability with RPC-over-RDMA Version 1 61 Implementations . . . . . . . . . . . . . . . . . . . 7 62 4.1.2. Interoperability Amongst RDMA Transports . . . . . . 7 63 5. Updating the Message Format . . . . . . . . . . . . . . . . . 7 64 5.1. Feature Support Flags . . . . . . . . . . . . . . . . . . 8 65 5.2. Inline Threshold Values . . . . . . . . . . . . . . . . . 8 66 6. Security Considerations . . . . . . . . . . . . . . . . . . . 9 67 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 68 7.1. Guidance for Designated Experts . . . . . . . . . . . . . 10 69 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 10 70 8.1. Normative References . . . . . . . . . . . . . . . . . . 10 71 8.2. Informative References . . . . . . . . . . . . . . . . . 11 72 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 12 73 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 75 1. Introduction 77 The RPC-over-RDMA version 1 transport protocol [RFC8166] enables 78 payload data transfer using Remote Direct Memory Access (RDMA) for 79 upper-layer protocols based on Remote Procedure Calls (RPC) 80 [RFC5531]. The terms "Remote Direct Memory Access" (RDMA) and 81 "Direct Data Placement" (DDP) are introduced in [RFC5040]. 83 The two most immediate shortcomings of RPC-over-RDMA version 1 are: 85 o Setting up an RDMA data transfer (via RDMA Read or Write) can be 86 costly. The small default size of messages transmitted using RDMA 87 Send forces the use of RDMA Read or Write operations even for 88 relatively small messages and data payloads. 89 The original specification of RPC-over-RDMA version 1 provided an 90 out-of-band protocol for passing inline threshold values between 91 connected peers [RFC5666]. However, [RFC8166] eliminated support 92 for this protocol making it unavailable for this purpose. 94 o Unlike most other contemporary RDMA-enabled storage protocols, 95 there is no facility in RPC-over-RDMA version 1 that enables the 96 use of remote invalidation [RFC5042]. 98 RPC-over-RDMA version 1 has no means of extending its XDR definition 99 in such a way that interoperability with existing implementations is 100 preserved. As a result, an out-of-band mechanism is needed to help 101 relieve these constraints for existing RPC-over-RDMA version 1 102 implementations. 104 This document specifies a simple, non-XDR-based message format 105 designed to be passed between RPC-over-RDMA version 1 peers at the 106 time each RDMA transport connection is first established. The 107 mechanism assumes that the underlying RDMA transport has a private 108 data field that is passed between peers at connection time, such as 109 is present in the iWARP protocol (described in Section 7.1 of 110 [RFC5044]) or the InfiniBand Connection Manager [IBA]. 112 To enable current RPC-over-RDMA version 1 implementations to 113 interoperate with implementations that support the private message 114 format described in this document, implementation of the private data 115 message is OPTIONAL. When the private data message has been 116 successfully exchanged, peers may choose to perform extended RDMA 117 semantics. However, the private message format does not alter the 118 XDR definition specified in [RFC8166]. 120 The message format is intended to be further extensible within the 121 normal scope of such IETF work (see Section 5 for further details). 122 Section 7 of the current document defines an IANA registry for this 123 purpose. In addition, interoperation between implementations of RPC- 124 over-RDMA version 1 that present this message format to peers and 125 those that do not recognize this message format is guaranteed. 127 2. Requirements Language 129 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 130 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 131 "OPTIONAL" in this document are to be interpreted as described in BCP 132 14 [RFC2119] [RFC8174] when, and only when, they appear in all 133 capitals, as shown here. 135 3. Advertised Transport Properties 137 3.1. Inline Threshold Size 139 Section 3.3.2 of [RFC8166] defines the term "inline threshold." An 140 inline threshold is the maximum number of bytes that can be 141 transmitted using one RDMA Send and one RDMA Receive. There are a 142 pair of inline thresholds for a connection: a client-to-server 143 threshold and a server-to-client threshold. 145 If an incoming RDMA message exceeds the size of a receiver's inline 146 threshold, the receive operation fails and the RDMA provider 147 typically terminates the connection. To convey an RPC message larger 148 than the receiver's inline threshold without risking receive failure, 149 a sender must use explicit RDMA data transfer operations, which are 150 more expensive than an RDMA Send. See Sections 3.3 and 3.5 of 151 [RFC8166] for a complete discussion. 153 The default value of inline thresholds for RPC-over-RDMA version 1 154 connections is 1024 bytes (as defined in Section 3.3.3 of [RFC8166]). 155 This value is adequate for nearly all NFS version 3 procedures. 157 NFS version 4 COMPOUND operations [RFC7530] are larger on average 158 than NFS version 3 procedures [RFC1813], forcing clients to use 159 explicit RDMA operations for frequently-issued requests such as 160 LOOKUP and GETATTR. The use of RPCSEC_GSS security also increases 161 the average size of RPC messages, due to the larger size of 162 RPCSEC_GSS credential material included in RPC headers [RFC7861]. 164 If a sender and receiver could somehow agree on larger inline 165 thresholds, frequently-used RPC transactions avoid the cost of 166 explicit RDMA operations. 168 3.2. Remote Invalidation 170 After an RDMA data transfer operation completes, an RDMA consumer can 171 use remote invalidation to request that the remote peer RNIC 172 invalidate an STag associated with the data transfer [RFC5042]. 174 An RDMA consumer requests remote invalidation by posting an RDMA Send 175 With Invalidate Work Request in place of an RDMA Send Work Request. 176 Each RDMA Send With Invalidate carries one STag to invalidate. The 177 receiver of an RDMA Send With Invalidate performs the requested 178 invalidation and then reports that invalidation as part of the 179 completion of a waiting Receive Work Request. 181 If both peers support remote invalidation, an RPC-over-RDMA responder 182 might use remote invalidation when replying to an RPC request that 183 provided chunks. Because one of the chunks has already been 184 invalidated, finalizing the results of the RPC is made simpler and 185 faster. 187 However, there are some important caveats which contraindicate the 188 blanket use of remote invalidation: 190 o Remote invalidation is not supported by all RNICs. 192 o Not all RPC-over-RDMA responder implementations can generate RDMA 193 Send With Invalidate Work Requests. 195 o Not all RPC-over-RDMA requester implementations can recognize when 196 remote invalidation has occurred. 198 o On one connection in different RPC-over-RDMA transactions, or in a 199 single RPC-over-RDMA transaction, an RPC-over-RDMA requester can 200 expose a mixture of STags that may be invalidated remotely and 201 some that must not be. No indication is provided at the RDMA 202 layer as to which is which. 204 A responder therefore must not employ remote invalidation unless it 205 is aware of support for it in its own RDMA stack, and on the 206 requester. And, without altering the XDR structure of RPC-over-RDMA 207 version 1 messages, it is not possible to support remote invalidation 208 with requesters that mix STags that may and must not be invalidated 209 remotely in a single RPC or on the same connection. 211 There are some NFS/RDMA client implementations whose STags are always 212 safe to invalidate remotely. For such clients, indicating to the 213 responder that remote invalidation is always safe can allow such 214 invalidation without the need for additional protocol to be defined. 216 4. Private Data Message Format 218 With an InfiniBand lower layer, for example, RDMA connection setup 219 uses a Connection Manager when establishing a Reliable Connection 220 [IBA]. When an RPC-over-RDMA version 1 transport connection is 221 established, the client (which actively establishes connections) and 222 the server (which passively accepts connections) populate the CM 223 Private Data field exchanged as part of CM connection establishment. 225 The transport properties exchanged via this mechanism are fixed for 226 the life of the connection. Each new connection presents an 227 opportunity for a fresh exchange. An implementation of the extension 228 described in this document MUST be prepared for the settings to 229 change upon a reconnection. 231 For RPC-over-RDMA version 1, the CM Private Data field is formatted 232 as described in the following subsection. RPC clients and servers 233 use the same format. If the capacity of the Private Data field is 234 too small to contain this message format, the underlying RDMA 235 transport is not managed by a Connection Manager, or the underlying 236 RDMA transport uses Private Data for its own purposes, the CM Private 237 Data field cannot be used on behalf of RPC-over-RDMA version 1. 239 The first 8 octets of the CM Private Data field is to be formatted as 240 follows: 242 0 1 2 3 243 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 244 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 245 | Format Identifier | 246 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 247 | Version | Flags | Send Size | Receive Size | 248 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 250 Format Identifier: This field contains a fixed 32-bit value that 251 identifies the content of the Private Data field as an RPC-over- 252 RDMA version 1 CM Private Data message. In RPC-over-RDMA version 253 1 Private Data, the value of this field is always 0xf6ab0e18, in 254 network byte order. The use of this field is further expanded 255 upon in Section 4.1.2. 257 Version: This 8-bit field contains a message format version number. 258 The value "1" in this field indicates that exactly eight octets 259 are present, that they appear in the order described in this 260 section, and that each has the meaning defined in this section. 261 Further considerations about the use of this field are discussed 262 in Section 5. 264 Flags: This 8-bit field contains bit flags that indicate the support 265 status of optional features, such as remote invalidation. The 266 meaning of these flags is defined in Section 5.1. 268 Send Size: This 8-bit field contains an encoded value corresponding 269 to the maximum number of bytes this peer is prepared to transmit 270 in a single RDMA Send on this connection. The value is encoded as 271 described in Section 5.2. 273 Receive Size: This 8-bit field contains an encoded value 274 corresponding to the maximum number of bytes this peer is prepared 275 to receive with a single RDMA Receive on this connection. The 276 value is encoded as described in Section 5.2. 278 4.1. Interoperability Considerations 280 The extension described in this document is designed to allow RPC- 281 over-RDMA version implementations that use CM Private Data to 282 interoperate fully with RPC-over-RDMA version 1 implementations that 283 do not exchange this information. Implementations that use this 284 extension must also interoperate fully with RDMA implementations that 285 use CM Private Data for other purposes. Realizing these goals 286 require that implementations of this extension follow the practices 287 described in the rest of this section. 289 4.1.1. Interoperability with RPC-over-RDMA Version 1 Implementations 291 When a peer does not receive a CM Private Data message which conforms 292 to Section 4, it needs to act as if the remote peer supports only the 293 default RPC-over-RDMA version 1 settings, as defined in [RFC8166]. 294 In other words, the peer MUST behave as if a Private Data message was 295 received in which bit 15 of the Flags field is zero, and both Size 296 fields contain the value zero. 298 4.1.2. Interoperability Amongst RDMA Transports 300 The Format Identifier field defined in Section 4 is provided to 301 enable implementations to distinguish RPC-over-RDMA version 1 Private 302 Data from private data inserted at other layers, such as the private 303 data inserted by the iWARP MPAv2 enhancement described in [RFC6581]. 305 As part of connection establishment, the received private data buffer 306 is searched for the Format Identifier word. The offset of the Format 307 Identifier is not restricted to any alignment. If the RPC-over-RDMA 308 version 1 CM Private Data Format Identifier is not present, an RPC- 309 over-RDMA version 1 receiver MUST behave as if no RPC-over-RDMA 310 version 1 CM Private Data has been provided. 312 Once the RPC-over-RDMA version 1 CM Private Data Format Identifier is 313 found, the receiver parses the subsequent octets as RPC-over-RDMA 314 version 1 CM Private Data. As additional assurance that the private 315 data content is valid RPC-over-RDMA version 1 CM Private Data, the 316 receiver should check that the format version number field contains a 317 valid and recognized version number, the size of the private data 318 does not overrun the length of the buffer, and all reserved flag bits 319 are zero. 321 5. Updating the Message Format 323 Although the message format described in this document provides the 324 ability for the client and server to exchange particular information 325 about the local RPC-over-RDMA implementation, it is possible that 326 there will be a future need to exchange additional properties. This 327 would make it necessary to extend or otherwise modify the format 328 described in this document. 330 Any modification faces the problem of interoperating properly with 331 implementations of RPC-over-RDMA version 1 that are unaware of this 332 existence of the new format. These include implementations that that 333 do not recognize the exchange of CM Private Data as well as those 334 that recognize only the format described in this document. 336 Given the message format described in this document, these 337 interoperability constraints could be met by the following sorts of 338 new message formats: 340 o A format which uses a different value for the first four bytes of 341 the format, as provided for in the registry described in 342 Section 7. 344 o A format which uses the same value for the Format Identifier field 345 and a value other than one (1) in the Version field. 347 Although it is possible to reorganize the last three of the eight 348 bytes in the existing format, extended formats are unlikely to do so. 349 New formats would take the form of extensions of the format described 350 in this document with added fields starting at byte eight of the 351 format and changes to the definition of previously reserved flags. 353 5.1. Feature Support Flags 355 The bits in the Flags field are labeled from bit 8 to bit 15, as 356 shown in the diagram above. When the Version field contains the 357 value "1", the bits in the Flags field are to be set as follows: 359 Bit 15: When both connection peers have set this flag in their CM 360 Private Data, the responder MAY use RDMA Send With Invalidate when 361 transmitting RPC Replies. Each RDMA Send With Invalidate MUST 362 invalidate an STag associated only with the XID in the rdma_xid 363 field of the RPC-over-RDMA Transport Header it carries. 364 When either peer on a connection clears this flag, the responder 365 MUST use only RDMA Send when transmitting RPC Replies. 367 Bits 14 - 8: These bits are reserved and are always zero when the 368 Version field contains 1. 370 5.2. Inline Threshold Values 372 Inline threshold sizes from 1KB to 256KB can be represented in the 373 Send Size and Receive Size fields. A sender computes the encoded 374 value by dividing the buffer size, in octets, by 1024 and subtracting 375 one from the result. A receiver decodes this value by performing the 376 inverse set of operations: it adds one to the encoded value and then 377 multiplies that result by 1024. 379 The client uses the smaller of its own send size and the server's 380 reported receive size as the client-to-server inline threshold. The 381 server uses the smaller of its own send size and the clients's 382 reported receive size as the server-to-client inline threshold. 384 6. Security Considerations 386 The reader is directed to the Security Considerations section of 387 [RFC8166] for background and further discussion. 389 The RPC-over-RDMA version 1 protocol framework depends on the 390 semantics of the Reliable Connected (RC) queue pair (QP) type, as 391 defined in Section 9.7.7 of [IBA]. The integrity of CM Private Data 392 and the authenticity of its source are ensured by the exclusive use 393 of RC queue pairs. Any attempt to interfere with or hijack data in 394 transit on an RC connection results in the RDMA provider terminating 395 the connection. 397 Additional analysis of RDMA transport security appears in the 398 Security Considerations section of [RFC5042]. That document 399 recommends IPsec as the default transport layer security solution. 400 When deployed with iWARP, IPsec establishes a protected channel 401 before any iWARP operations are exchanged, thus it protects the 402 exchange of Private Data that occurs as each QP is established. 403 However, IPsec is not available for InfiniBand or RoCE deployments. 404 Those fabrics rely on physical security and cyclic redundancy checks 405 to protect network traffic. 407 Improperly setting one of the fields in a version 1 Private Message 408 can result in an increased risk of disconnection (i.e., self-imposed 409 Denial of Service). There is no additional risk of exposing upper- 410 layer payloads after exchanging the Private Message format defined in 411 the current document. 413 In addition to describing the structure of a new format version, any 414 document that extends the Private Data format described in the 415 current document must discuss security considerations of new data 416 items exchanged between connection peers. 418 7. IANA Considerations 420 In accordance with [RFC8126], the author requests that IANA create a 421 new registry in the "Remote Direct Data Placement" Protocol Category 422 Group. The new registry is to be called the "RDMA-CM Private Data 423 Identifier Registry". This is a registry of 32-bit numbers that 424 identify the upper-layer protocol associated with data that appears 425 in the application-specific RDMA-CM Private Data area. The fields in 426 this registry include: Format Identifier, Description, and Reference. 428 The initial contents of this registry are a single entry: 430 +------------------+------------------------------------+-----------+ 431 | Format | Format Description | Reference | 432 | Identifier | | | 433 +------------------+------------------------------------+-----------+ 434 | 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] | 435 | | Data | | 436 +------------------+------------------------------------+-----------+ 438 Table 1: RDMA-CM Private Data Identifier Registry 440 IANA is to assign subsequent new entries in this registry using the 441 Expert Review policy as defined in Section 4.5 of [RFC8126]. 443 7.1. Guidance for Designated Experts 445 The Designated Expert (DE), appointed by the IESG, should ascertain 446 the existence of suitable documentation that defines the semantics 447 and format of the private data, and verify that the document is 448 permanently and publicly available. Documentation produced outside 449 the IETF must not conflict with work that is active or already 450 published within the IETF. 452 The new Reference field should contain a reference to that 453 documentation. The DE can assign new Format Identifiers at random as 454 long as they do not conflict with existing entries in this registry. 455 The Description field should contain the name of the RDMA consumer 456 that will generate and use the private data. 458 The DE will post the request to the nfsv4 WG mailing list (or a 459 successor to that list, if such a list exists), for comment and 460 review. The DE will approve or deny the request and publish notice 461 of the decision within 30 days. 463 8. References 465 8.1. Normative References 467 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 468 Requirement Levels", BCP 14, RFC 2119, 469 DOI 10.17487/RFC2119, March 1997, 470 . 472 [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. 473 Garcia, "A Remote Direct Memory Access Protocol 474 Specification", RFC 5040, DOI 10.17487/RFC5040, October 475 2007, . 477 [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement 478 Protocol (DDP) / Remote Direct Memory Access Protocol 479 (RDMAP) Security", RFC 5042, DOI 10.17487/RFC5042, October 480 2007, . 482 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 483 Writing an IANA Considerations Section in RFCs", BCP 26, 484 RFC 8126, DOI 10.17487/RFC8126, June 2017, 485 . 487 [RFC8166] Lever, C., Ed., Simpson, W., and T. Talpey, "Remote Direct 488 Memory Access Transport for Remote Procedure Call Version 489 1", RFC 8166, DOI 10.17487/RFC8166, June 2017, 490 . 492 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 493 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 494 May 2017, . 496 8.2. Informative References 498 [IBA] InfiniBand Trade Association, "InfiniBand Architecture 499 Specification Volume 1", Release 1.3, March 2015. 501 Available from https://www.infinibandta.org/ 503 [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS 504 Version 3 Protocol Specification", RFC 1813, 505 DOI 10.17487/RFC1813, June 1995, 506 . 508 [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. 509 Carrier, "Marker PDU Aligned Framing for TCP 510 Specification", RFC 5044, DOI 10.17487/RFC5044, October 511 2007, . 513 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 514 Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, 515 May 2009, . 517 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 518 Transport for Remote Procedure Call", RFC 5666, 519 DOI 10.17487/RFC5666, January 2010, 520 . 522 [RFC6581] Kanevsky, A., Ed., Bestler, C., Ed., Sharp, R., and S. 523 Wise, "Enhanced Remote Direct Memory Access (RDMA) 524 Connection Establishment", RFC 6581, DOI 10.17487/RFC6581, 525 April 2012, . 527 [RFC7530] Haynes, T., Ed. and D. Noveck, Ed., "Network File System 528 (NFS) Version 4 Protocol", RFC 7530, DOI 10.17487/RFC7530, 529 March 2015, . 531 [RFC7861] Adamson, A. and N. Williams, "Remote Procedure Call (RPC) 532 Security Version 3", RFC 7861, DOI 10.17487/RFC7861, 533 November 2016, . 535 Acknowledgments 537 Thanks to Christoph Hellwig and Devesh Sharma for suggesting this 538 approach, and to Tom Talpey and Dave Noveck for their expert comments 539 and review. The author also wishes to thank Bill Baker and Greg 540 Marsden for their support of this work. Also, thanks to expert 541 reviewers Sean Hefty and Dave Minturn. 543 Special thanks go to document shepherd Brian Pawlowski, Transport 544 Area Director Magnus Westerlund, NFSV4 Working Group Chairs David 545 Noveck and Spencer Shepler, and NFSV4 Working Group Secretary Thomas 546 Haynes. 548 Author's Address 550 Charles Lever 551 Oracle Corporation 552 United States of America 554 Email: chuck.lever@oracle.com