idnits 2.17.1 draft-cel-nfsv4-rpcrdma-version-two-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 3, 2016) is 2883 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-11) exists of draft-ietf-nfsv4-rfc5666bis-07 == Outdated reference: A later version (-08) exists of draft-ietf-nfsv4-rpcrdma-bidirection-04 -- Obsolete informational reference (is this intentional?): RFC 5661 (Obsoleted by RFC 8881) -- Obsolete informational reference (is this intentional?): RFC 5666 (Obsoleted by RFC 8166) Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network File System Version 4 C. Lever, Ed. 3 Internet-Draft Oracle 4 Intended status: Standards Track D. Noveck 5 Expires: December 5, 2016 HPE 6 June 3, 2016 8 RPC-over-RDMA Version Two Protocol 9 draft-cel-nfsv4-rpcrdma-version-two-01 11 Abstract 13 This document specifies an improved protocol for conveying Remote 14 Procedure Call (RPC) messages on physical transports capable of 15 Remote Direct Memory Access (RDMA), based on RPC-over-RDMA Version 16 One. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on December 5, 2016. 35 Copyright Notice 37 Copyright (c) 2016 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 54 2. Inline Threshold . . . . . . . . . . . . . . . . . . . . . . 3 55 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 56 2.2. Motivation . . . . . . . . . . . . . . . . . . . . . . . 4 57 2.3. Default Values . . . . . . . . . . . . . . . . . . . . . 4 58 3. Protocol Extensibility . . . . . . . . . . . . . . . . . . . 5 59 3.1. Optional Features . . . . . . . . . . . . . . . . . . . . 5 60 3.2. Message Direction . . . . . . . . . . . . . . . . . . . . 5 61 3.3. Documentation Requirements . . . . . . . . . . . . . . . 6 62 4. XDR Protocol Definition . . . . . . . . . . . . . . . . . . . 7 63 4.1. Code Component License . . . . . . . . . . . . . . . . . 8 64 4.2. RPC-Over-RDMA Version Two XDR . . . . . . . . . . . . . . 10 65 5. Protocol Version Negotiation . . . . . . . . . . . . . . . . 13 66 5.1. Responder Does Support RPC-over-RDMA Version Two . . . . 13 67 5.2. Responder Does Not Support RPC-over-RDMA Version Two . . 13 68 5.3. Requester Does Not Support RPC-over-RDMA Version Two . . 14 69 6. Security Considerations . . . . . . . . . . . . . . . . . . . 14 70 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 71 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 14 72 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 73 9.1. Normative References . . . . . . . . . . . . . . . . . . 14 74 9.2. Informative References . . . . . . . . . . . . . . . . . 15 75 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 16 77 1. Introduction 79 Remote Direct Memory Access (RDMA) [RFC5040] [RFC5041] [IB] is a 80 technique for moving data efficiently between end nodes. By 81 directing data into destination buffers as it is sent on a network, 82 and placing it via direct memory access by hardware, the 83 complementary benefits of faster transfers and reduced host overhead 84 are obtained. 86 A protocol already exists that enables ONC RPC [RFC5531] messages to 87 be conveyed on RDMA transports. That protocol is RPC-over-RDMA 88 Version One, specified in [I-D.ietf-nfsv4-rfc5666bis]. RPC-over-RDMA 89 Version One is deployed and in use, though there are some 90 shortcomings to this protocol, such as: 92 o The use of small Receive buffers force the use of RDMA Read and 93 Write transfers for small payloads, and limit the size of 94 backchannel messages 96 o Lack of support for potential optimizations, such as remote 97 invalidation, that require changes to on-the-wire behavior 99 To address these issues in a way that is compatible with existing 100 RPC-over-RDMA Version One deployments, a new version of RPC-over-RDMA 101 is presented in this document. RPC-over-RDMA Version Two contains 102 only incremental changes over RPC-over-RDMA Version One to facilitate 103 adoption of Version Two by existing Version One implementations. 105 The major new feature in RPC-over-RDMA Version Two is extensibility 106 of the RPC-over-RDMA header. Extensibility enables narrow changes to 107 RPC-over-RDMA Version Two so that new optional capabilities can be 108 introduced without a protocol version change and while maintaining 109 interoperability with existing implementations. New capabilities can 110 be proposed and developed independently of each other, and 111 implementaters can choose among them. It should be straightforward 112 to create and document experimental features and then bring them 113 through the standards process. 115 In addition to extensibility, the default inline threshold value is 116 larger in RPC-over-RDMA Version Two. This change is driven by the 117 increase in average size of RPC messages containing common NFS 118 operations. With NFSv4.1 [RFC5661] and later, compound operations 119 convey more data per RPC message. The default 1KB inline threshold 120 in RPC-over-RDMA Version One prevents attaining the best possible 121 performance. 123 1.1. Requirements Language 125 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 126 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 127 document are to be interpreted as described in [RFC2119]. 129 2. Inline Threshold 131 2.1. Terminology 133 The term "inline threshold" is defined in Section 4 of 134 [I-D.ietf-nfsv4-rfc5666bis]. An "inline threshold" value is the 135 largest message size (in octets) that can be conveyed in one 136 direction between peer implementations using RDMA Send and Receive. 137 Each connection has two inline threshold values: one for messages 138 flowing from requester-to-responder (referred to as the "call inline 139 threshold"), and one for messages flowing from responder-to-requester 140 (referred to as the "reply inline threshold"). Inline threshold 141 values are not advertised to peers via the base RPC-over-RDMA Version 142 Two protocol. 144 A connection's inline threshold determines when full RDMA Read or 145 Write operations are required because the RPC message to be send is 146 larger than the peer's Receive buffer. If an RPC message does not 147 contain DDP-eligible data items, a requester prepares a Long Call or 148 Reply to convey the whole RPC message using an RDMA Read or Write. 150 2.2. Motivation 152 RDMA Read and Write operations require that each data payload resides 153 in a region of memory that is registered with the RNIC. When an RPC 154 is complete, that region is unregistered, fencing it from the 155 responder. 157 Both registration and unregistration have a latency cost which is 158 insignificant compared to data handling costs. When a data payload 159 is small, however, the cost of registering and unregistering the 160 memory where it resides becomes a relatively significant part of 161 total RPC latency. Therefore the most efficient operation of RPC- 162 over-RDMA occurs when RDMA Read and Write operations are used for 163 large payloads, and avoided for small payloads. 165 When RPC-over-RDMA Version One was conceived, the typical size of RPC 166 messages that did not involve a significant data payload was under 167 500 bytes. A 1024-byte inline threshold adequately minimized the 168 frequency of inefficient Long Calls and Replies. 170 Starting with NFSv4.1 [RFC5661], NFS COMPOUND RPC messages are larger 171 and more complex than before. With a 1024-byte inline threshold, 172 RDMA Read or Write operations are needed for frequent operations that 173 do not bear a data payload, such as GETATTR and LOOKUP, reducing the 174 efficiency of the transport. To reduce the need to use Long Calls 175 and Replies, RPC-over-RDMA Version Two quadruples the default inline 176 threshold size. This also increases the maximum size of backward 177 direction RPC messages. 179 2.3. Default Values 181 RPC-over-RDMA Version Two receiver implementations MUST support an 182 inline threshold of 4096 bytes, but MAY support larger inline 183 threshold values. A mechanism for discovering a peer's preferred 184 inline threshold value may be used to optimize RDMA Send operations 185 further. In the absense of such a mechanism, senders MUST assume a 186 receiver's inline threshold is 4096 bytes. 188 The new default inline threshold size is no larger than the size of a 189 hardware page on typical platforms. This conserves the resources 190 needed to Send and Receive base level RPC-over-RDMA Version Two 191 messages, enabling RPC-over-RDMA Version Two to be used on a broad 192 base of hardware. 194 3. Protocol Extensibility 196 The core RPC-over-RDMA Version Two header format is specified in 197 Section 4 as a complete and stand-alone piece of XDR. Any change to 198 this XDR description requires a protocol version number change. 200 3.1. Optional Features 202 RPC-over-RDMA Version Two introduces the ability to extend the core 203 protocol via optional features. Extensibility enables minor protocol 204 issues to be addressed and incremental enhancements to be made 205 without the need to change the protocol version. The key capability 206 is that both sides can detect whether a feature is supported by their 207 peer or not. With this ability, OPTIONAL features can be introduced 208 over time to an otherwise stable protocol. 210 The rdma_opttype field carries a 32-bit unsigned integer. The value 211 in this field denotes an optional operation that MAY be supported by 212 the receiver. The values of this field and their meaning are defined 213 in other Standards Track documents. 215 The rdma_optinfo field carries opaque data. The content of this 216 field is data meaningful to the optional operation denoted by the 217 value in rdma_opttype. The content of this field is not defined in 218 the base RPC-over-RDMA Version Two protocol, but is defined in other 219 Standards Track documents 221 When an implementation does not recognize or support the value 222 contained in the rdma_opttype field, it MUST send an RPC-over-RDMA 223 message with the rdma_xid field set to the same value as the 224 erroneous message, the rdma_proc field set to RDMA2_ERROR, and the 225 rdma_err field set to RDMA2_ERR_INVAL_OPTION. 227 3.2. Message Direction 229 Backward direction operation depends on the ability of the receiver 230 to distinguish between incoming forward and backward direction calls 231 and replies. This needs to be done because both the XID field and 232 the flow control value (RPC-over-RDMA credits) in the RPC-over-RDMA 233 header are interpreted in the context of each message's direction. 235 A receiver typically distinguishes message direction by examining the 236 mtype field in the RPC header of each incoming payload message. 237 However, RDMA2_OPTIONAL type messages may not carry an RPC message 238 payload. 240 To enable RDMA2_OPTIONAL type messages that do not carry an RPC 241 message payload to be interpreted unambiguously, the rdma2_optional 242 structure contains a field that identifies the message direction. A 243 similar field has been added to the rpcrdma2_chunks and 244 rpcrdma2_error structures to simplify parsing the RPC-over-RDMA 245 header at the receiver. 247 3.3. Documentation Requirements 249 RPC-over-RDMA Version Two may be extended by defining a new 250 rdma_opttype value, and then by providing an XDR description of the 251 rdma_optinfo content that corresponds with the new rdma_opttype 252 value. As a result, a new header type is effectively created. 254 A Standards Track document introduces each set of such protocol 255 elements. Together these elements are considered an OPTIONAL 256 feature. Each implementation is either aware of all the protocol 257 elements introduced by that feature, or is aware of none of them. 259 Documents describing extensions to RPC-over-RDMA Version Two should 260 contain: 262 o An explanation of the purpose and use of each new protocol element 263 added 265 o An XDR description of the protocol elements, and a script to 266 extract it 268 o A mechanism for reporting errors when the error is outside the 269 available choices already available in the base protocol or in 270 other extensions 272 o An indication of whether a Payload stream must be present, and a 273 description of its contents 275 o A description of interactions with existing extensions 277 The last bullet includes requirements that another OPTIONAL feature 278 needs to be present for new protocol elements to work, or that a 279 particular level of support be provided for some particular facility 280 for the new extension to work. 282 Implementers combine the XDR descriptions of the new features they 283 intend to use with the XDR description of the base protocol in this 284 document. This may be necessary to create a valid XDR input file 285 because extensions are free to use XDR types defined in the base 286 protocol, and later extensions may use types defined by earlier 287 extensions. 289 The XDR description for the RPC-over-RDMA Version Two protocol 290 combined with that for any selected extensions should provide an 291 adequate human-readable description of the extended protocol. 293 4. XDR Protocol Definition 295 This section contains a description of the core features of the RPC- 296 over-RDMA Version Two protocol, expressed in the XDR language 297 [RFC4506]. 299 This description is provided in a way that makes it simple to extract 300 into ready-to-compile form. The reader can apply the following shell 301 script to this document to produce a machine-readable XDR description 302 of the RPC-over-RDMA Version One protocol without any OPTIONAL 303 extensions. 305 307 #!/bin/sh 308 grep '^ *///' | sed 's?^ /// ??' | sed 's?^ *///$??' 310 312 That is, if the above script is stored in a file called "extract.sh" 313 and this document is in a file called "spec.txt" then the reader can 314 do the following to extract an XDR description file: 316 318 sh extract.sh < spec.txt > rpcrdma_corev2.x 320 322 Optional extensions to RPC-over-RDMA Version Two, published as 323 Standards Track documents, will have similar means of providing XDR 324 that describes those extensions. Once XDR for all desired extensions 325 is also extracted, it can be appended to the XDR description file 326 extracted from this document to produce a consolidated XDR 327 description file reflecting all extensions selected for an RPC-over- 328 RDMA implementation. 330 4.1. Code Component License 332 Code components extracted from this document must include the 333 following license text. When the extracted XDR code is combined with 334 other complementary XDR code which itself has an identical license, 335 only a single copy of the license text need be preserved. 337 339 /// /* 340 /// * Copyright (c) 2010, 2016 IETF Trust and the persons 341 /// * identified as authors of the code. All rights reserved. 342 /// * 343 /// * The authors of the code are: 344 /// * B. Callaghan, T. Talpey, C. Lever, and D. Noveck. 345 /// * 346 /// * Redistribution and use in source and binary forms, with 347 /// * or without modification, are permitted provided that the 348 /// * following conditions are met: 349 /// * 350 /// * - Redistributions of source code must retain the above 351 /// * copyright notice, this list of conditions and the 352 /// * following disclaimer. 353 /// * 354 /// * - Redistributions in binary form must reproduce the above 355 /// * copyright notice, this list of conditions and the 356 /// * following disclaimer in the documentation and/or other 357 /// * materials provided with the distribution. 358 /// * 359 /// * - Neither the name of Internet Society, IETF or IETF 360 /// * Trust, nor the names of specific contributors, may be 361 /// * used to endorse or promote products derived from this 362 /// * software without specific prior written permission. 363 /// * 364 /// * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS 365 /// * AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED 366 /// * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 367 /// * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 368 /// * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 369 /// * EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 370 /// * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, 371 /// * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 372 /// * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 373 /// * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 374 /// * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 375 /// * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 376 /// * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 377 /// * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 378 /// * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 379 /// */ 381 383 4.2. RPC-Over-RDMA Version Two XDR 385 The XDR defined in this section is used to encode the Transport 386 Header Stream in each RPC-over-RDMA Version Two message. The terms 387 "Transport Header Stream" and "RPC Payload Stream" are defined in 388 Section 4 of [I-D.ietf-nfsv4-rfc5666bis]. 390 392 /// /* From RFC 5531, Section 9 */ 393 /// enum msg_type { 394 /// CALL = 0, 395 /// REPLY = 1 396 /// }; 397 /// 398 /// struct rpcrdma2_segment { 399 /// uint32 rdma_handle; 400 /// uint32 rdma_length; 401 /// uint64 rdma_offset; 402 /// }; 403 /// 404 /// struct rpcrdma2_read_segment { 405 /// uint32 rdma_position; 406 /// struct rpcrdma2_segment rdma_target; 407 /// }; 408 /// 409 /// struct rpcrdma2_read_list { 410 /// struct rpcrdma2_read_segment rdma_entry; 411 /// struct rpcrdma2_read_list *rdma_next; 412 /// }; 413 /// 414 /// struct rpcrdma2_write_chunk { 415 /// struct rpcrdma2_segment rdma_target<>; 416 /// }; 417 /// 418 /// struct rpcrdma2_write_list { 419 /// struct rpcrdma2_write_chunk rdma_entry; 420 /// struct rpcrdma2_write_list *rdma_next; 421 /// }; 422 /// 423 /// struct rpcrdma2_chunk_lists { 424 /// enum msg_type rdma_direction; 425 /// struct rpcrdma2_read_list *rdma_reads; 426 /// struct rpcrdma2_write_list *rdma_writes; 427 /// struct rpcrdma2_write_chunk *rdma_reply; 428 /// }; 429 /// 430 /// enum rpcrdma2_errcode { 431 /// RDMA2_ERR_VERS = 1, 432 /// RDMA2_ERR_BAD_HEADER = 2, 433 /// RDMA2_ERR_INVAL_OPTION = 3 434 /// }; 435 /// 436 /// struct rpcrdma2_err_vers { 437 /// uint32 rdma_vers_low; 438 /// uint32 rdma_vers_high; 439 /// }; 440 /// 441 /// union rpcrdma2_error switch (rpcrdma2_errcode rdma_err) { 442 /// case RDMA2_ERR_VERS: 443 /// rpcrdma2_err_vers rdma_vrange; 444 /// case RDMA2_ERR_BAD_HEADER: 445 /// void; 446 /// case RDMA2_ERR_INVAL_OPTION: 447 /// void; 448 /// }; 449 /// 450 /// struct rpcrdma2_optional { 451 /// enum msg_type rdma_optdir; 452 /// uint32 rdma_opttype; 453 /// opaque rdma_optinfo<>; 454 /// }; 455 /// 456 /// enum rpcrdma2_proc { 457 /// RDMA2_MSG = 0, 458 /// RDMA2_NOMSG = 1, 459 /// RDMA2_ERROR = 4, 460 /// RDMA2_OPTIONAL = 5 461 /// }; 462 /// 463 /// union rpcrdma2_body switch (rpcrdma2_proc rdma_proc) { 464 /// case RDMA2_MSG: 465 /// rpcrdma2_chunks rdma_chunks; 466 /// case RDMA2_NOMSG: 467 /// rpcrdma2_chunks rdma_chunks; 468 /// case RDMA2_ERROR: 469 /// rpcrdma2_error rdma_error; 470 /// case RDMA2_OPTIONAL: 471 /// rpcrdma2_optional rdma_optional; 472 /// }; 473 /// 474 /// struct rpcrdma2_xprt_hdr { 475 /// uint32 rdma_xid; 476 /// uint32 rdma_vers; 477 /// uint32 rdma_credit; 478 /// rpcrdma2_body rdma_body; 479 /// }; 481 483 4.2.1. Presence Of Payload 485 o When the rdma_proc field has the value RDMA2_MSG, an RPC Payload 486 Stream MUST follow the Transport Header Stream in the Send buffer. 488 o When the rdma_proc field has the value RDMA2_ERROR, an RPC Payload 489 Stream MUST NOT follow the Transport Header Stream. 491 o When the rdma_proc field has the value RDMA2_OPTIONAL, all, part 492 of, or no RPC Payload Stream MAY follow the Transport header 493 Stream in the Send buffer. 495 4.2.2. Message Direction 497 Implementations of RPC-over-RDMA Version Two are REQUIRED to support 498 backwards direction operation as described in 499 [I-D.ietf-nfsv4-rpcrdma-bidirection]. 501 o When the rdma_proc field has the value RDMA2_MSG or RDMA2_NOMSG, 502 the value of the rdma_direction field MUST be the same as the 503 value of the associated RPC message's msg_type field. 505 o When the rdma_proc field has the value RDMA2_ERROR, the direction 506 of the message is always Responder-to-Requester (REPLY). 508 o When the rdma_proc field has the value RDMA2_OPTIONAL and a whole 509 or partial RPC message payload is present, the value of the 510 rdma_optdir field MUST be the same as the value of the associated 511 RPC message's msg_type field. 513 o When the rdma_proc field has the value RDMA2_OPTIONAL and no RPC 514 message payload is present, a Requester MUST set the value of the 515 rdma_optdir field to CALL, and a Responder MUST set the value of 516 the rdma_optdir field to REPLY. The Requester chooses a value for 517 the rdma_xid field from the XID space that matches the message's 518 direction. Requesters and Responders set the rdma_credit field in 519 a similar fashion: a value is set that is appropriate for the 520 direction of the message. 522 4.2.3. Transport Errors 524 Error handling works the same way in RPC-over-RDMA Version Two as it 525 does in RPC-over-RDMA Version One, with one change described in 526 Section 3.1. Version One error handling is described in Section 5 of 527 [I-D.ietf-nfsv4-rfc5666bis]. 529 5. Protocol Version Negotiation 531 When an RPC-over-RDMA Version Two requester establishes a connection 532 to a responder, the first order of business is to determine the 533 responder's highest supported protocol version. 535 As with RPC-over-RDMA Version One, a requester MUST assume the 536 ability to exchange only a single RPC-over-RDMA message at a time 537 until it receives a non-error RPC-over-RDMA message from the 538 responder that reports the responder's actual credit limit. 540 First, the requester sends a single valid RPC-over-RDMA message with 541 the value two (2) in the rdma_vers field. Because the responder 542 might support only RPC-over-RDMA Version One, this initial message 543 can be no larger than the Version One default inline threshold of 544 1024 bytes. 546 5.1. Responder Does Support RPC-over-RDMA Version Two 548 If the responder does support RPC-over-RDMA Version Two, it sends an 549 RPC-over-RDMA message back to the requester with the same XID 550 containing a valid non-error response. Subsequently, both peers use 551 the default inline threshold value for RPC-over-RDMA Version Two 552 connections (4096 bytes). 554 5.2. Responder Does Not Support RPC-over-RDMA Version Two 556 If the responder does not support RPC-over-RDMA Version Two, 557 [I-D.ietf-nfsv4-rfc5666bis] REQUIRES that it send an RPC-over-RDMA 558 message to the requester with the same XID, with RDMA2_ERROR in the 559 rdma_proc field, and with the error code RDMA2_ERR_VERS. This 560 message also reports a range of protocol versions that the responder 561 supports. To continue operation, the requester selects a protocol 562 version in the range of responder-supported versions for subsequent 563 messages on this connection. 565 If the connection is lost immediately after the RDMA2_ERROR reply is 566 received, a requester can avoid a possible version negotiation loop 567 when re-establishing another connection by assuming that particular 568 responder does not support RPC-over-RDMA Version Two. A requester 569 can assume the same situation (no responder support for RPC-over-RDMA 570 Version Two) if the initial negotiation message is lost or dropped. 572 Once the negotiation exchange is complete, both peers use the default 573 inline threshold value for the protocol version that will be used for 574 the remainder of the connection lifetime. To permit inline threshold 575 values to change during negotiation of protocol version, RPC-over- 576 RDMA Version Two implementations MUST allow inline threshold values 577 to change without triggering a connection loss. 579 5.3. Requester Does Not Support RPC-over-RDMA Version Two 581 [I-D.ietf-nfsv4-rfc5666bis] REQUIRES that a responder MUST send 582 Replies with the same RPC-over-RDMA protocol version that the 583 requester uses to send its Calls. 585 6. Security Considerations 587 The security considerations for RPC-over-RDMA Version Two are the 588 same as those for RPC-over-RDMA Version One. 590 7. IANA Considerations 592 There are no IANA considerations at this time. 594 8. Acknowledgments 596 The authors gratefully acknowledge the work of Brent Callaghan and 597 Tom Talpey on the original RPC-over-RDMA Version One specification 598 [RFC5666]. The authors also wish to thank Bill Baker, Greg Marsden, 599 and Matt Benjamin for their support of this work. 601 The extract.sh shell script and formatting conventions were first 602 described by the authors of the NFSv4.1 XDR specification [RFC5662]. 604 Special thanks go to nfsv4 Working Group Chair Spencer Shepler and 605 nfsv4 Working Group Secretary Thomas Haynes for their support. 607 9. References 609 9.1. Normative References 611 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 612 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 613 RFC2119, March 1997, 614 . 616 [RFC4506] Eisler, M., Ed., "XDR: External Data Representation 617 Standard", STD 67, RFC 4506, DOI 10.17487/RFC4506, May 618 2006, . 620 [RFC5531] Thurlow, R., "RPC: Remote Procedure Call Protocol 621 Specification Version 2", RFC 5531, DOI 10.17487/RFC5531, 622 May 2009, . 624 9.2. Informative References 626 [I-D.ietf-nfsv4-rfc5666bis] 627 Lever, C., Simpson, W., and T. Talpey, "Remote Direct 628 Memory Access Transport for Remote Procedure Call, Version 629 One", draft-ietf-nfsv4-rfc5666bis-07 (work in progress), 630 May 2016. 632 [I-D.ietf-nfsv4-rpcrdma-bidirection] 633 Lever, C., "Bi-directional Remote Procedure Call On RPC- 634 over-RDMA Transports", draft-ietf-nfsv4-rpcrdma- 635 bidirection-04 (work in progress), May 2016. 637 [IB] InfiniBand Trade Association, "InfiniBand Architecture 638 Specifications", . 640 [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. 641 Garcia, "A Remote Direct Memory Access Protocol 642 Specification", RFC 5040, DOI 10.17487/RFC5040, October 643 2007, . 645 [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct 646 Data Placement over Reliable Transports", RFC 5041, DOI 647 10.17487/RFC5041, October 2007, 648 . 650 [RFC5661] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 651 "Network File System (NFS) Version 4 Minor Version 1 652 Protocol", RFC 5661, DOI 10.17487/RFC5661, January 2010, 653 . 655 [RFC5662] Shepler, S., Ed., Eisler, M., Ed., and D. Noveck, Ed., 656 "Network File System (NFS) Version 4 Minor Version 1 657 External Data Representation Standard (XDR) Description", 658 RFC 5662, DOI 10.17487/RFC5662, January 2010, 659 . 661 [RFC5666] Talpey, T. and B. Callaghan, "Remote Direct Memory Access 662 Transport for Remote Procedure Call", RFC 5666, DOI 663 10.17487/RFC5666, January 2010, 664 . 666 Authors' Addresses 668 Charles Lever (editor) 669 Oracle Corporation 670 1015 Granger Avenue 671 Ann Arbor, MI 48104 672 USA 674 Phone: +1 734 274 2396 675 Email: chuck.lever@oracle.com 677 David Noveck 678 Hewlett Packard Enterprise 679 165 Dascomb Road 680 Andover, MA 01810 681 USA 683 Phone: +1 978 474 2011 684 Email: davenoveck@gmail.com