idnits 2.17.1 draft-ietf-storm-rdmap-ext-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There are 4 instances of too long lines in the document, the longest one being 2 characters in excess of 72. ** The abstract seems to contain references ([RFC5040]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The exact meaning of the all-uppercase expression 'MAY NOT' is not defined in RFC 2119. If it is intended as a requirements expression, it should be rewritten using one of the combinations defined in RFC 2119; otherwise it should not be all-uppercase. == The expression 'MAY NOT', while looking like RFC 2119 requirements text, is not defined in RFC 2119, and should not be used. Consider using 'MUST NOT' instead (if that is what you mean). Found 'MAY NOT' in this paragraph: Implementation Note: Implementations MAY NOT guarantee Atomicity if the Tagged Buffer is accessed by any other method other than an Atomic Operation within the scope of a single RNIC. -- The document date (September 5, 2013) is 3885 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFCXXXX' is mentioned on line 1243, but not defined == Missing Reference: 'RFC5226' is mentioned on line 1241, but not defined ** Obsolete undefined reference: RFC 5226 (Obsoleted by RFC 8126) == Unused Reference: 'RFC5042' is defined on line 1259, but no explicit reference was found in the text Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Storage Maintenance (storm) Working Group Hemal Shah 2 Internet Draft Broadcom Corporation 3 Intended status: Standards Track Felix Marti 4 Expires: March 2014 Wael Noureddine 5 Asgeir Eiriksson 6 Chelsio Communications, Inc. 7 Robert Sharp 8 Intel Corporation 9 September 5, 2013 11 RDMA Protocol Extensions 12 draft-ietf-storm-rdmap-ext-05.txt 14 Status of this Memo 16 This Internet-Draft is submitted to IETF in full conformance with 17 the provisions of BCP 78 and BCP 79. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF). Note that other groups may also distribute 21 working documents as Internet-Drafts. The list of current Internet- 22 Drafts is at http://datatracker.ietf.org/drafts/current. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as "work in progress." 29 This Internet-Draft will expire on March 5, 2014. 31 Copyright Notice 33 Copyright (c) 2013 IETF Trust and the persons identified as the 34 document authors. All rights reserved. 36 This document is subject to BCP 78 and the IETF Trust's Legal 37 Provisions Relating to IETF Documents 38 (http://trustee.ietf.org/license-info) in effect on the date of 39 publication of this document. Please review these documents 40 carefully, as they describe your rights and restrictions with 41 respect to this document. Code Components extracted from this 42 document must include Simplified BSD License text as described in 43 Section 4.e of the Trust Legal Provisions and are provided without 44 warranty as described in the Simplified BSD License. 46 Abstract 48 This document specifies extensions to the IETF Remote Direct Memory 49 Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write 50 services directly to applications and enables data to be transferred 51 directly into Upper Layer Protocol (ULP) Buffers without 52 intermediate data copies. The extensions specified in this document 53 provide the following capabilities and/or improvements: Atomic 54 Operations and Immediate Data. 56 Table of Contents 58 1. Introduction...................................................3 59 2. Requirements Language..........................................4 60 3. Glossary.......................................................4 61 4. Header Format Extensions.......................................7 62 4.1. RDMAP Control and Invalidate STag Fields..................7 63 4.2. RDMA Message Definitions..................................9 64 5. Atomic Operations..............................................9 65 5.1. Atomic Operation Details.................................11 66 5.1.1. FetchAdd............................................11 67 5.1.2. Swap................................................12 68 5.1.3. CmpSwap.............................................13 69 5.2. Atomic Operations........................................15 70 5.2.1. Atomic Operation Request Message....................15 71 5.2.2. Atomic Operation Response Message...................19 72 5.3. Atomicity Guarantees.....................................21 73 5.4. Atomic Operations Ordering and Completion Rules..........21 74 6. Immediate Data................................................22 75 6.1. RDMAP Interactions with ULP for Immediate Data...........22 76 6.2. Immediate Data Header Format.............................23 77 6.3. Immediate Data or Immediate Data with SE Message.........23 78 6.4. Ordering and Completions.................................24 79 7. Ordering and Completions Table................................24 80 8. Error Processing..............................................27 81 8.1. Errors Detected at the Local Peer........................28 82 8.2. Errors Detected at the Remote Peer.......................28 84 9. Security Considerations.......................................29 85 10. IANA Considerations..........................................29 86 10.1. RDMAP Message Atomic Operation Subcodes.................29 87 11. References...................................................31 88 11.1. Normative References....................................31 89 11.2. Informative References..................................31 90 12. Acknowledgments..............................................32 91 Appendix A. DDP Segment Formats for RDMA Messages................33 92 A.1. DDP Segment for Atomic Operation Request.................33 93 A.2. DDP Segment for Atomic Response..........................35 94 A.3. DDP Segment for Immediate Data and Immediate Data with SE35 96 1. Introduction 98 The RDMA Protocol [RFC5040] provides capabilities for zero copy data 99 communications that preserve memory protection semantics, enabling 100 more efficient network protocol implementations. This document 101 specifies the following extensions to the RDMA Protocol (RDMAP): 103 o Atomic operations on remote memory locations. Support for atomic 104 operation enhances the usability of RDMAP in distributed shared 105 memory environments. 107 o Immediate Data messages allow the ULP at the sender to provide a 108 small amount of data. When an Immediate Data message is sent 109 following an RDMA Write Message, the combination of the two 110 messages is an implementation of RDMA Write with Immediate 111 message that is found in other RDMA transport protocols. 113 Other RDMA transport protocols define the functionality added by 114 these extensions leading to differences in RDMA applications and/or 115 Upper Layer Protocols. Removing these differences in the transport 116 protocols simplifies these applications and ULPs and that is the 117 main motivation for the extensions specified in this document. 119 [RSOCKETS] is an example of RDMA enabled middleware that provides a 120 socket interface as the upper edge interface and utilizes RDMA to 121 provide more efficient networking for sockets based applications. 122 [RSOCKETS] is aware of Immediate Data support in [IB]. [RSOCKETS] 123 cannot utilize the RDMA Write with Immediate Data operation from 124 [IB] on iWARP. The addition of the Immediate Data operation 125 specified in this draft will alleviate this difference in [RSOCKETS] 126 when running on [IB] and iWARP. 128 [DAT_ATOMICS] is an example of RDMA enable middleware that provides 129 a portable RDMA programming interface for various RDMA transport 130 protocols. [DAT_ATOMICS] includes a primitive for [IB] that is not 131 supported by iWARP RNICs. The addition of Atomic Operations as 132 specified in this draft will allow atomic operations in 133 [DAT_ATOMICS] to work for both [IB] and iWARP interchangeably. 135 1.1. Discovery of RDMAP Extensions 137 Today there are RDMA applications and/or ULPs that are aware of the 138 existence of Atomic and Immediate data operations for RDMA 139 transports such as [IB]. Today, these applications need to be aware 140 that iWARP RNICs do not support these operations. Typically the 141 availability of these capabilities are exposed to the applications 142 through adapter query interfaces in software. Applications then 143 have to decide to use or not to use Immediate Data or Atomic 144 Operations based on the results of the query interfaces. 145 Negotiation of Atomic Operations typically are to determine the 146 scope of atomicity guarantees, not down to the individual Atomic 147 Operations supported. Therefore, this specification requires all 148 Atomic Operations defined within to be supported if an RNIC supports 149 any Atomic Operations. In cases where heterogeneous hardware, with 150 differing support for Atomic Operations and Immediate Data 151 Operations, is deployed for usage by RDMA applications and/or ULPs, 152 applications are either statically configured to use or not use 153 optional features or use application specific negotiation 154 mechanisms. For the extensions covered by this document, it is 155 RECOMMENDED that RDMA applications and/or ULPs negotiate at the 156 application or ULP level the usage of these extensions. The 157 definition of such application specific mechanism is outside the 158 scope of this specification. For backward compatibility, existing 159 applications and/or ULPs should assume that iWARP RNICs do not 160 support these extensions. 162 2. Requirements Language 164 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 165 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 166 document are to be interpreted as described in RFC-2119 [RFC2119]. 168 3. Glossary 170 This document is an extension of [RFC5040] and key words are defined 171 in the glossary of the referenced document. 173 Atomic Operation - is an operation that results in an execution of a 174 memory operation at a specific ULP Buffer address on a remote node 175 using the Tagged Buffer data transfer model. The consumer can use 176 Atomic Operations to read, modify and write memory at the 177 destination ULP Buffer address while at the same time guarantee that 178 no other Atomic Operation read or write accesses to the ULP Buffer 179 address targeted by the Atomic Operation will occur across any other 180 RDMAP Streams on an RNIC at the Responder. Atomicity guarantees 181 across multiple RNICs or between other applications/ULPs running on 182 the Responder with access to the ULP Buffer are outside the scope of 183 this specification. 185 Atomic Operation Request - An RDMA Message used by the Data Source 186 to perform an Atomic Operation at the Responder. 188 Atomic Operation Response - An RDMA Message used by the Responder to 189 describe the completion of an Atomic Operation at the Responder. 191 CmpSwap - is an Atomic Operation that is used to compare and swap a 192 value at a specific address on a remote node. 194 FetchAdd - is an Atomic Operation that is used to atomically 195 increment a value at a specific ULP Buffer address on a remote node. 197 Immediate Data - a small fixed size portion of data sent from the 198 Data Source to a Data Sink 200 Immediate Data Message - An RDMA Message used by the Data Source to 201 send Immediate Data to the Data Sink 203 Immediate Data with Solicited Event (SE) Message - An RDMA Message 204 used by the Data Source to send Immediate Data with Solicited Event 205 to the Data Sink 207 Requester - the sender of an RDMA Atomic Operation request. 209 Responder - the receiver of an RDMA Atomic Operation request. 211 Swap - is an Atomic Operation that is used to swap a value at a 212 specific address on a remote node. 214 ULP - Upper Level Protocol. The protocol layer above the one 215 currently being referenced. The ULP for RDMAP[RFC5040]/DDP[RFC5041] 216 is expected to be an OS, Application, adaptation layer, or 217 proprietary device. The RDMAP[RFC5040]/DDP[RFC5041] documents do not 218 specify a ULP -- they provide a set of semantics that allow a ULP to 219 be designed to utilize RDMAP[RFC5040]/DDP[RFC5041]. 221 4. Header Format Extensions 223 The control information of RDMA Messages is included in DDP protocol 224 [RFC5041] defined header fields. [RFC5040] defines the RDMAP header 225 formats layered on the [RFC5041] DDP header definition. This 226 specification extends [RFC5040] with the following new formats: 227 . Four new RDMA Messages carry additional RDMAP headers. The 228 Immediate Data operation and Immediate Data with Solicited Event 229 operation include 8 bytes of data following the RDMAP header. 230 Atomic Operations include Atomic Request or Atomic Response 231 headers following the RDMAP header. The RDMAP header for Atomic 232 Request messages is 52 bytes long as specified in Figure 4. The 233 RDMAP header for Atomic Response Messages is 32 bytes long as 234 specified in Figure 5. 236 . Introduction of a new queue for untagged buffers (QN=3) used for 237 Atomic Response tracking. 239 4.1. RDMAP Control and Invalidate STag Fields 241 For reference, Figure 1 depicts the format of the DDP Control and 242 RDMAP Control fields, in the style and convention of [RFC5040]: 244 0 1 2 3 245 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 246 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 247 |T|L| Resrv | DV| RV|Rsv| Opcode| 248 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 249 | Invalidate STag | 250 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 252 Figure 1 DDP Control and RDMAP Control Fields 254 The DDP Control Field consists of the T,L, Resrv and DV fields 255 [RFC5041]. The RDMAP Control Field consists of the RV, Rsv and 256 Opcode fields [RFC5040]. 258 This specification adds additional values for the RDMA Opcode field 259 to those specified in [RFC5040]. Figure 2 defines the new values of 260 RDMA Opcode field that MUST be used for the RDMA Messages defined in 261 this specification. 263 Figure 2 also defines when the STag, Tagged Offset, and Queue Number 264 fields MUST be provided for the RDMA Messages defined in this 265 specification. 267 All RDMA Messages defined in this specification MUST have: 269 The RDMA Version (RV) field: 01b. 271 Opcode field: See Figure 2. 273 Invalidate STag: MUST be set to zero by the sender, ignored by the 274 receiver. 276 -------+-----------+-------+------+-------+-----------+-------------- 277 RDMA | Message | Tagged| STag | Queue | Invalidate| Message 278 Opcode | Type | Flag | and | Number| STag | Length 279 | | | TO | | | Communicated 280 | | | | | | between DDP 281 | | | | | | and RDMAP 282 -------+-----------+-------+------+-------+-----------+-------------- 283 1000b | Immediate | 0 | N/A | 0 | N/A | Yes 284 | Data | | | | | 285 -------+-----------+------------------------------------------------- 286 1001b | Immediate | 0 | N/A | 0 | N/A | Yes 287 | Data with | | | | | 288 | SE | | | | | 289 -------+-----------+------------------------------------------------- 290 1010b | Atomic | 0 | N/A | 1 | N/A | Yes 291 | Request | | | | | 292 -------+-----------+------------------------------------------------- 293 1011b | Atomic | 0 | N/A | 3 | N/A | Yes 294 | Response | | | | | 295 -------+-----------+------------------------------------------------- 297 Figure 2 Additional RDMA Usage of DDP Fields 299 Note: N/A means Not Applicable. 301 This extension defines RDMAP use Queue Number 3 for Untagged Buffers 302 for Atomic Responses. This queue is used for tracking outstanding 303 Atomic Requests. 305 All other DDP and RDMAP control fields MUST be set as described in 306 [RFC5040]. 308 4.2. RDMA Message Definitions 310 The following figure defines which RDMA Headers MUST be used on each 311 new RDMA Message and which new RDMA Messages are allowed to carry 312 ULP payload: 314 -------+-----------+-------------------+------------------------- 315 RDMA | Message | RDMA Header Used | ULP Message allowed in 316 Message| Type | | the RDMA Message 317 OpCode | | | 318 | | | 319 -------+-----------+-------------------+------------------------- 320 1000b | Immediate | Immediate Data | No 321 | Data | Header | 322 -------+-----------+-------------------+------------------------- 323 1001b | Immediate | Immediate Data | No 324 | Data with | Header | 325 | SE | | 326 -------+-----------+-------------------+------------------------- 327 1010b | Atomic | Atomic Request | No 328 | Request | Header | 329 -------+-----------+-------------------+------------------------- 330 1011b | Atomic | Atomic Response | No 331 | Response | Header | 332 -------+-----------+-------------------+------------------------- 333 Figure 3 RDMA Message Definitions 335 5. Atomic Operations 337 The RDMA Protocol Specification in [RFC5040] does not include 338 support for Atomic Operations which are an important building block 339 for implementing distributed shared memory. 341 This document extends the RDMA Protocol specification with a set of 342 basic Atomic Operations, and specifies their resource and ordering 343 rules. The Atomic Operations specified in this document provide 344 equivalent functionality to the [IB] RDMA transport, to allow 345 applications that use these primitives to work interchangeably over 346 iWARP. In addition, this document specifies a basic Swap operation. 347 Other operations are left for future consideration. 349 Atomic operations as specified in this document execute a 64-bit 350 memory operation at a specified destination ULP Buffer address on a 351 Responder node using the Tagged Buffer data transfer model. The 352 operations atomically read, modify and write back the contents of 353 the destination ULP Buffer address and guarantee that Atomic 354 Operations on this ULP Buffer address by other RDMAP Streams on the 355 same RNIC do not occur between the read and the write caused by the 356 Atomic Operation. Therefore, the Responder RNIC MUST implement 357 mechanisms to prevent Atomic Operations to a memory registered for 358 Atomic Operations while an Atomic Operation targeting the memory is 359 in progress. Atomicity guarantees between multiple RNICs or between 360 RNICs and software running independent of the RNIC is outside the 361 scope of this specification. An RNIC that supports Atomic 362 Operations as specified in this document MUST implement all Atomic 363 Operation Codes defined in Figure 5. The advertisement of Tagged 364 Buffer information for Atomic Operations is outside the scope of 365 this specification and must be handled by the ULPs. 367 Implementation note: It is recommended that the applications do not 368 use the ULP Buffer addresses used for Atomic Operations for other 369 RDMA operations. 371 Implementation note: Errors related to the alignment in the 372 following sections cover Atomic Operations targeted at a ULP Buffer 373 address that is not aligned to a 64-bit boundary. 375 Atomic Operation Request Messages use the same remote addressing 376 mechanism as RDMA Reads and Writes. The ULP Buffer address specified 377 in the request is in the address space of the Remote Peer to which 378 the Atomic Operation is targeted. 380 Atomic Operation Response Messages MUST use the Untagged Buffer 381 model with QN=3. Queue number 3 MUST be used to track outstanding 382 Atomic Operation Request messages at the Requestor. When the Atomic 383 Operation Response message is received, the MSN MUST be used to 384 locate the corresponding Atomic Operation request in order to 385 complete the Atomic Operation request. 387 5.1. Atomic Operation Details 389 The following sub-sections describe the Atomic Operations in more 390 details. 392 5.1.1. FetchAdd 394 The FetchAdd Atomic Operation requests the Responder to read a 64- 395 bit Original Remote Data Value at a 64-bit aligned ULP Buffer 396 address in the Responder's memory, to perform FetchAdd operation on 397 multiple fields of selectable length specified by 64-bit "Add Mask", 398 and write the result back to the same ULP Buffer address. The Atomic 399 addition is performed independently on each one of these fields. A 400 bit set in the Add Mask field specifies the field boundary. FetchAdd 401 Atomic Operations MUST target ULP Buffer addresses that are 64-bit 402 aligned. FetchAdd Atomic Operations that target ULP Buffer addresses 403 that are not 64-bit aligned MUST be surfaced as errors and the 404 Responder's memory MUST NOT be modified in such cases. Additionally 405 an error MUST be surfaced and a terminate message MUST be generated. 406 The setting of "Add Mask" field to 0x0000000000000000 results in 407 Atomic Add of 64-bit Original Remote Data Value and 64-bit "Add 408 Data". 410 The pseudo code below describes masked FetchAdd Atomic Operation. 412 bit_location = 1 414 carry = 0 416 Remote Data Value = 0 418 for bit = 0 to 63 420 { 422 if (bit != 0 ) bit_location = bit_location << 1 424 val1 = !(!(Original Remote Data Value & bit_location)) 426 val2 = !(!(Add Data & bit_location)) 428 sum = carry + val1 + val2 430 carry = !(!(sum & 2)) 431 sum = sum & 1 433 if (sum) 435 Remote Data Value |= bit_location 437 carry = ((carry) && (!(Add Mask & bit_location))) 439 } 441 The FetchAdd operation is performed in the endian format of the 442 target memory. The "Original Remote Data Value" is converted from 443 the endian format of the target memory for return and returned to 444 the Requester. The fields are in big-endian format on the wire. 446 The Requester specifies: 448 o Remote STag 450 o Remote Tagged Offset 452 o Add Data 454 o Add Mask 456 The Responder returns: 458 o Original Remote Data 460 5.1.2. Swap 462 The Swap Atomic Operation requires the Responder to read a 64-bit 463 value at a 64-bit aligned ULP Buffer address in the Responder's 464 memory, then to write the "Swap Data" fields into the same ULP 465 Buffer address. The "Original Remote Data Value" is converted from 466 the endian format of the target memory for return and returned to 467 the Requester. The fields are in big-endian format on the wire. 469 The Requester specifies: 471 o Remote STag 473 o Remote Tagged Offset 475 o Swap Data 476 The Responder returns: 478 o Original Remote Data Value 480 After the successful completion of Swap operation, the Responder's 481 memory at the specified ULP Buffer address MUST contain the "Swap 482 Data" field in the header. Swap Atomic Operations MUST target ULP 483 Buffer addresses that are 64-bit aligned. Swap Atomic Operations 484 that target ULP Buffer addresses that are not 64-bit aligned: 486 o The operation MUST NOT be performed, 488 o The Responder's memory MUST NOT be modified, 490 o The result MUST be surfaced as an error, and 492 o A terminate message MUST be generated (see Section 8.2. for the 493 terminate message contents) 495 5.1.3. CmpSwap 497 The CmpSwap Atomic Operation requires the Responder to read a 64-bit 498 value at a 64-bit aligned ULP Buffer address in the Responder's 499 memory, to perform an AND logical operation using the 64 bit 500 "Compare Mask" field in the Atomic Operation Request header, then to 501 compare it with the result of a logical AND operation of the 502 "Compare Mask" and the "Compare Data" fields in the header, and, if 503 the two values are equal, to swap masked bits in the same ULP Buffer 504 address with the masked Swap Data. If the two masked compare values 505 are not equal, the contents of the Responder's memory are not 506 changed. In either case, the original value read from the ULP Buffer 507 address is converted from the endian format of the target memory for 508 return and returned to the Requester. The fields are in big-endian 509 format on the wire. 511 The Requester specifies: 513 o Remote STag 515 o Remote Tagged Offset 517 o Swap Data 518 o Swap Mask 520 o Compare Data 522 o Compare Mask 524 The Responder returns: 526 o Original Remote Data Value 528 The following pseudo code describes the masked CmpSwap operation 529 result. 531 if (!((Compare Data ^ Original Remote Data Value) & Compare Mask) 533 then 535 Remote Data Value = 537 (Original Remote Data Value & ~(Swap Mask)) 539 | (Swap Data & Swap Mask) 541 else 543 Remote Data Value = Original Remote Data Value 545 After the operation, the remote data buffer MUST contain the 546 "Original Remote Data Value" (if comparison did not match) or the 547 masked "Swap Data" (if the comparison did match). CmpSwap Atomic 548 Operations MUST target buffer addresses that are 64-bit aligned. 549 CmpSwap Atomic Operations that target buffer addresses that are not 550 64-bit aligned: 552 o The operation MUST NOT be performed, 554 o The Responder's memory MUST NOT be modified, 556 o The result MUST be surfaced as an error, and 558 o A terminate message MUST be generated (see Section 8.2. for the 559 terminate message contents) 561 5.2. Atomic Operations 563 The Atomic Operation Request and Response are RDMA Messages. An 564 Atomic Operation makes use of the DDP Untagged Buffer Model. Atomic 565 Operation Request messages MUST use the same Queue Number as RDMA 566 Read Requests (QN=1). Reusing the same Queue Number for Atomic 567 Request messages allows the Atomic Operations to reuse the same 568 infrastructure (e.g. ORD/IRD flow control) as defined for RDMA Read 569 Requests. Atomic Operation Response messages MUST set Queue Number 570 (QN) to 3 in the DDP header. 572 The RDMA Message OpCode for an Atomic Request Message is 1010b. The 573 RDMA Message OpCode for an Atomic Response Message is 1011b. 575 5.2.1. Atomic Operation Request Message 577 The Atomic Operation Request Message carries an Atomic Operation 578 Header that describes the ULP Buffer address in the Responder's 579 memory. The Atomic Operation Request header immediately follows the 580 DDP header. The RDMAP layer passes to the DDP layer a RDMAP Control 581 Field. The following figure depicts the Atomic Operation Request 582 Header that MUST be used for all Atomic Operation Request Messages: 584 0 1 2 3 585 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 586 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 587 | Reserved (Not Used) |AOpCode| 588 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 589 | Request Identifier | 590 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 591 | Remote STag | 592 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 593 | Remote Tagged Offset | 594 + + 595 | | 596 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 597 | Add or Swap Data | 598 + + 599 | | 600 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 601 | Add or Swap Mask | 602 + + 603 | | 604 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 605 | Compare Data | 606 + + 607 | | 608 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 609 | Compare Mask | 610 + + 611 | | 612 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 614 Figure 4 Atomic Operation Request Header 616 Reserved (Not Used): 28 bits 618 This field MUST be set to zero on transmit, ignored on 619 receive. 621 Atomic Operation Code (AOpCode): 4 bits. 623 See Figure 5. All Atomic Operation Codes from Figure 5 MUST 624 be implemented by an RNIC that support Atomic Operations. 626 Request Identifier: 32 bits. 628 The Request Identifier specifies a number that is used to 629 identify Atomic Operation Request Message. The value used in 630 this field is implementation dependent and outside the scope 631 of this specification. The value of this field is reflected 632 back to the Local Peer in the Atomic Operation Response 633 message. 635 Remote STag: 32 bits. 637 The Remote STag identifies the Remote Peer's Tagged Buffer 638 targeted by the Atomic Operation. The Remote STag is 639 associated with the RDMAP Stream through a mechanism that is 640 outside the scope of the RDMAP specification. 642 Remote Tagged Offset: 64 bits. 644 The Remote Tagged Offset specifies the starting offset, in 645 octets, from the base of the Remote Peer's Tagged Buffer 646 targeted by the Atomic Operation. The Remote Tagged Offset MAY 647 start at an arbitrary offset. 649 Add or Swap Data: 64 bits. 651 The Add or Swap Data field specifies the 64-bit "Add Data" 652 value in an Atomic FetchAdd Operation or the 64-bit "Swap 653 Data" value in an Atomic Swap or CmpSwap Operation. 655 Add or Swap Mask: 64 bits 657 This field is used in masked Atomic Operations (FetchAdd and 658 CmpSwap) to perform a bitwise logical AND operation as 659 specified in the definition of these operations. For non- 660 masked Atomic Operations (Swap), this field MUST be set to 661 ffffffffffffffffh on transmit and ignored by the receiver. 663 Compare Data: 64 bits. 665 The Compare Data field specifies the 64-bit "Compare Data" 666 value in an Atomic CmpSwap Operation. For Atomic FetchAdd and 667 Atomic Swap operation, the Compare Data field MUST be set to 668 zero on transmit and ignored by the receiver. 670 Compare Mask: 64 bits 671 This field is used in masked Atomic Operation CmpSwap to 672 perform a bitwise logical AND operation as specified in the 673 definition of these operations. For Atomic Operations FetchAdd 674 and Swap, this field MUST be set to ffffffffffffffffh on 675 transmit and ignored by the receiver. 677 ---------+-----------+----------+----------+---------+--------- 678 Atomic | Atomic | Add or | Add or | Compare | Compare 679 Operation| Operation | Swap | Swap | Data | Mask 680 Code | | Data | Mask | | 681 ---------+-----------+----------+----------+---------+--------- 682 0000b | FetchAdd | Add Data | Add Mask | N/A | N/A 683 ---------+-----------+----------+----------+---------+--------- 684 0001b | Swap | Swap Data| N/A | N/A | N/A 685 ---------+-----------+----------+----------+---------+--------- 686 0010b | CmpSwap | Swap Data| Swap Mask| Valid | Valid 687 ---------+-----------+----------+----------+---------+--------- 688 0011b | | 689 to | Reserved | Not Specified 690 1111b | | 691 ---------+-----------+----------------------------------------- 693 Figure 5 Atomic Operation Message Definitions 695 The Atomic Operation Request Message has the following semantics: 697 1. An Atomic Operation Request Message MUST reference an Untagged 698 Buffer. That is, the Local Peer's RDMAP layer MUST request that 699 the DDP mark the Message as Untagged. 701 2. One Atomic Operation Request Message MUST consume one Untagged 702 Buffer. 704 3. The Responder's RDMAP layer MUST process an Atomic Operation 705 Request Message. A valid Atomic Operation Request Message MUST 706 NOT be delivered to the Responder's ULP (i.e., it is processed by 707 the RDMAP layer). 709 4. At the Responder, when an invalid Atomic Operation Request 710 Message with a reserved Atomic Operation Code is delivered to the 711 Remote Peer's RDMAP layer, an error MUST be surfaced. 713 5. An Atomic Operation Request Message MUST reference the RDMA Read 714 Request Queue. That is, the Requester's RDMAP layer MUST request 715 that the DDP layer set the Queue Number field to one. 717 6. The Requester MUST pass to the DDP layer Atomic Operation Request 718 Messages in the order they were submitted by the ULP. 720 7. The Responder MUST process the Atomic Operation Request Messages 721 in the order they were sent. 723 8. If the Responder receives a valid Atomic Operation Request 724 Message, it MUST respond with a valid Atomic Operation Response 725 Message. 727 5.2.2. Atomic Operation Response Message 729 The Atomic Operation Response Message carries an Atomic Operation 730 Response Header that contains the "Original Request Identifier" and 731 "Original Remote Data Value". The Atomic Operation Response Header 732 immediately follows the DDP header. The RDMAP layer passes to the 733 DDP layer a RDMAP Control Field. The following figure depicts the 734 Atomic Operation Response header that MUST be used for all Atomic 735 Operation Response Messages: 737 0 1 2 3 738 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 739 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 740 | Original Request Identifier | 741 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 742 | Original Remote Data Value | 743 + + 744 | | 745 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 747 Figure 6 Atomic Operation Response Header 749 Original Request Identifier: 32 bits. 751 The Original Request Identifier MUST be set to the value 752 specified in the Request Identifier field that was originally 753 provided in the corresponding Atomic Operation Request 754 Message. 756 Original Remote Data Value: 64 bits. 758 The Original Remote Value specifies the original 64-bit value 759 stored at the ULP Buffer address targeted by the Atomic 760 Operation. 762 The Atomic Operation Response Message has the following semantics: 764 1. The Atomic Operation Response Message for the associated Atomic 765 Operation Request Message travels in the opposite direction. 767 2. An Atomic Operation Response Message MUST consume an Untagged 768 Buffer. That is, the Responder RDMAP layer MUST request that the 769 DDP mark the Message as Untagged. 771 3. An Atomic Operation Response Message MUST reference the Queue 772 Number 3. That is, the Responder's RDMAP layer MUST request that 773 the DDP layer set the Queue Number field to 3. 775 4. The Responder MUST ensure that a sufficient number of Untagged 776 Buffers are available on the RDMA Read Request Queue (Queue with 777 DDP Queue Number 1) to support the maximum number of Atomic 778 Operation Requests negotiated by the ULP in addition to the 779 maximum number of RDMA Read Requests negotiated by the ULP. 781 5. The Requester MUST ensure that a sufficient number of Untagged 782 Buffers are available on the RDMA Atomic Response Queue (Queue 783 with DDP Queue Number 3) to support the maximum number of Atomic 784 Operation Requests negotiated by the ULP. 786 6. The RDMAP layer MUST Deliver the Atomic Operation Response 787 Message to the ULP. 789 7. At the Requester, when an invalid Atomic Operation Response 790 Message is delivered to the Remote Peer's RDMAP layer, an error 791 is surfaced. 793 8. When the Responder receives Atomic Operation Request messages, 794 the Responder RDMAP layer MUST pass Atomic Operation Response 795 Messages to the DDP layer, in the order that the Atomic Operation 796 Request Messages were received by the RDMAP layer, at the 797 Responder. 799 5.3. Atomicity Guarantees 801 Atomicity of the Read-Modify-Write (RMW) on the Responder's node by 802 the Atomic Operation MUST be assured in the context of concurrent 803 atomic accesses by other RDMAP Streams on the same RNIC. 805 5.4. Atomic Operations Ordering and Completion Rules 807 In addition to the ordering and completion rules described in 808 [RFC5040], the following rules apply to implementations of the 809 Atomic operations. 811 1. For an Atomic operation, the Requester MUST NOT consider the 812 contents of the Tagged Buffer at the Responder to be modified by 813 that specific Atomic Operation until the Atomic Operation 814 Response Message has been Delivered to RDMAP at the Requester. 816 2. Atomicity guarantees MUST be within the scope of a single RNIC. 818 Implementation Note: Implementations MAY NOT guarantee Atomicity 819 if the Tagged Buffer is accessed by any other method other than 820 an Atomic Operation within the scope of a single RNIC. 822 3. Atomic Operation Request Messages MUST NOT start processing at 823 the Responder until they have been Delivered to RDMAP by DDP. 825 4. Atomic Operation Response Messages MAY be generated at the 826 Responder after subsequent RDMA Write Messages or Send Messages 827 have been Placed or Delivered. 829 5. Atomic Operation Response Message processing at the Responder 830 MUST be started only after the Atomic Operation Request Message 831 has been Delivered by the DDP layer (thus, all previous RDMA 832 Messages on that DDP Stream have been Delivered). 834 6. Send Messages MAY be Completed at the Responder before prior 835 incoming Atomic Operation Request Messages have completed their 836 response processing. 838 7. An Atomic Operation MUST NOT be Completed at the Requester until 839 the DDP layer Delivers the associated incoming Atomic Operation 840 Response Message. 842 8. If more than one outstanding Atomic Request Messages are 843 supported by both peers, the Atomic Operation Request Messages 844 MUST be processed in the order they were delivered by the DDP 845 layer on the Responder. Atomic Operation Response Messages MUST 846 be submitted to the DDP layer on the Responder in the order the 847 Atomic Operation Request Messages were Delivered by DDP. 849 6. Immediate Data 851 The Immediate Data operation is typically used in conjunction with an 852 RDMA Write Operation to improve ULP processing efficiency. The 853 efficiency is gained by causing an RDMA Completion to be generated 854 immediately following the RDMA Write operation. This RDMA Completion 855 delivers 8 bytes of immediate data at the Remote Peer. The combination 856 of an RDMA Write Message followed by an Immediate Data Operation has 857 the same behavior as the RDMA Write with Immediate Data operation found 858 in [IB]. An Immediate Data operation that is not preceded by an RDMA 859 Write operation causes an RDMA Completion. 861 6.1. RDMAP Interactions with ULP for Immediate Data 863 For Immediate Data operations, the following are the interactions 864 between the RDMAP Layer and the ULP: 865 . At the Data Source: 867 . The ULP passes to the RDMAP Layer the following: 869 . Eight bytes of ULP Immediate Data 871 . When the Immediate Data operation Completes, an indication 872 of the Completion results. 874 . At the Data Sink: 876 . If the Immediate Data operation is Completed successfully, 877 the RDMAP Layer passes the following information to the ULP 878 Layer: 880 . Eight bytes of Immediate Data 882 . An Event, if the Data Sink is configured to generate an 883 Event. 885 . If the Immediate Data operation is Completed in error, the 886 Data Sink RDMAP Layer will pass up the corresponding error 887 information to the Data Sink ULP and send a Terminate 888 Message to the Data Source RDMAP Layer. The Data Source 889 RDMAP Layer will then pass up the Terminate Message to the 890 ULP. 892 6.2. Immediate Data Header Format 894 The Immediate Data and Immediate Data with SE Messages carry immediate 895 data as shown in Figure 7. The RDMAP layer passes to the DDP layer an 896 RDMAP Control Field and 8 bytes of Immediate Data. The first 8 bytes 897 of the data following the DDP header contains the Immediate Data. See 898 section A.3. for the DDP segment format of an Immediate Data or 899 Immediate Data with SE Message. 901 0 1 2 3 902 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 903 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 904 | Immediate Data | 905 + + 906 | | 907 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 909 Figure 7 Immediate Data or Immediate Data with SE Message Header 911 Immediate Data: 64 bits. 912 Eight bytes of data transferred from the Requester to an 913 untagged buffer at the Responder. 915 6.3. Immediate Data or Immediate Data with SE Message 917 The Immediate Data or Immediate Data with SE Message uses the DDP 918 Untagged Buffer Model to transfer Immediate Data from the Data 919 Source to the Data Sink. 920 . An Immediate Data or Immediate Data with SE Message MUST 921 reference an Untagged Buffer. That is, the Local Peer's RDMAP 922 Layer MUST request that the DDP layer mark the Message as 923 Untagged. 925 . One Immediate Data or Immediate Data with SE Message MUST consume 926 one Untagged Buffer. 928 . At the Remote Peer, the Immediate Data or Immediate Data with SE 929 Message MUST be Delivered to the Remote Peer's ULP in the order 930 they were sent. 932 . For an Immediate Data or Immediate Data with SE Message, the 933 Local Peer's RDMAP Layer MUST request that the DDP layer set the 934 Queue Number field to zero. 936 . For an Immediate Data or Immediate Data with SE Message, the 937 Local Peer's RDMAP Layer MUST request that the DDP layer transmit 938 8 bytes of data. 940 . The Local Peer MUST issue Immediate Data and Immediate Data with 941 SE Messages in the order they were submitted by the ULP. 943 . The Remote Peer MUST check that Immediate Data and Immediate Data 944 with SE Messages include exactly 8 bytes of data from the DDP 945 layer. The DDP header carries the length field that is reported 946 by the DDP layer. 948 6.4. Ordering and Completions 950 Ordering and completion rules for Immediate Data are the same as 951 those for a Send operation as described in section 5.5 of RFC 5040. 953 7. Ordering and Completions Table 955 The following table summarizes the ordering relationships for Atomic 956 and Immediate Data operations from the standpoint of Local Peer issuing 957 the Operations. Note that in the table that follows, Send includes 958 Send, Send with Invalidate, Send with Solicited Event, and Send with 959 Solicited Event and Invalidate. Also note that in the table below, 960 Immediate Data includes Immediate Data and Immediate Data with 961 Solicited Event. 963 ----------+------------+-------------+-------------+------------------- 964 First | Second | Placement | Placement | Ordering 965 Operation | Operation | Guarantee at| Guarantee at| Guarantee at 966 | | Remote Peer | Local Peer | Remote Peer 967 ----------+------------+-------------+-------------+------------------- 968 Immediate | Send | No Placement| Not | Completed in 969 Data | | Guarantee | Applicable | Order 970 | | between Send| | 971 | | Payload and | | 972 | | Immediate | | 973 | | Data | | 974 ----------+------------+-------------+-------------+------------------- 975 Immediate | RDMA | No Placement| Not | Not 976 Data | Write | Guarantee | Applicable | Applicable 977 | | between RDMA| | 978 | | Write | | 979 | | Payload and | | 980 | | Immediate | | 981 | | Data | | 982 ----------+------------+-------------+-------------+------------------- 983 Immediate | RDMA | No Placement| RDMA Read | RDMA Read 984 Data | Read | Guarantee | Response | Response 985 | | between | will not be | Message will 986 | | Immediate | Placed until| not be 987 | | Data and | Immediate | generated 988 | | RDMA Read | Data is | until 989 | | Request | Placed at | Immediate Data 990 | | | Remote Peer | has been 991 | | | | Completed 992 ----------+------------+-------------+-------------+------------------- 993 Immediate | Atomic | No Placement| Atomic | Atomic 994 Data | | Guarantee | Response | Response 995 | | between | will not be | Message will 996 | | Immediate | Placed until| not be 997 | | Data and | Immediate | generated 998 | | Atomic | Data is | until 999 | | Request | Placed at | Immediate Data 1000 | | | Remote Peer | has been 1001 | | | | Completed 1002 ----------+------------+-------------+-------------+------------------- 1003 Immediate | Immediate | No Placement| Not | Completed in 1004 Data or | Data | Guarantee | Applicable | Order 1005 Send | | | | 1006 ----------+------------+-------------+-------------+------------------- 1007 RDMA Write| Immediate | No Placement| Not | Immediate Data 1008 | Data | Guarantee | Applicable | is Completed 1009 | | | | after RDMA 1010 | | | | Write is Placed 1011 | | | | and Delivered 1012 ----------+------------+-------------+-------------+------------------- 1013 RDMA Read | Immediate | No Placement| Immediate | Not Applicable 1014 | Data | Guarantee | Data MAY be | 1015 | | between | Placed | 1016 | | Immediate | before | 1017 | | Data and | RDMA Read | 1018 | | RDMA Read | Response is | 1019 | | Request | generated | 1020 ----------+------------+-------------+-------------+------------------- 1021 Atomic | Immediate | No Placement| Immediate | Not Applicable 1022 | Data | Guarantee | Data MAY be | 1023 | | between | Placed | 1024 | | Immediate | before | 1025 | | Data and | Atomic | 1026 | | Atomic | Response is | 1027 | | Request | generated | 1028 ----------+------------+-------------+-------------+------------------- 1029 Atomic | Send | No Placement| Send Payload| Not Applicable 1030 | | Guarantee | MAY be | 1031 | | between Send| Placed | 1032 | | Payload and | before | 1033 | | Atomic | Atomic | 1034 | | Request | Response is | 1035 | | | generated | 1036 ----------+------------+-------------+-------------+------------------- 1037 Atomic | RDMA | No Placement| RDMA Write | Not 1038 | Write | Guarantee | Payload MAY | Applicable 1039 | | between RDMA| be Placed | 1040 | | Write | before | 1041 | | Payload and | Atomic | 1042 | | Atomic | Response is | 1043 | | Request | generated | 1044 ----------+------------+-------------+-------------+------------------- 1045 Atomic | RDMA | No Placement| No Placement| RDMA Read 1046 | Read | Guarantee | Guarantee | Response 1047 | | between | between | Message will 1048 | | Atomic | Atomic | not be 1049 | | Request and | Response | generated 1050 | | RDMA Read | and RDMA | until Atomic 1051 | | Request | Read | Response Message 1052 | | | Response | has been 1053 | | | | generated 1054 ----------+------------+-------------+-------------+------------------- 1055 Atomic | Atomic | Placed in | No Placement| Second Atomic 1056 | | order | Guarantee | Request 1057 | | | between two | Message will 1058 | | | Atomic | not be 1059 | | | Responses | processed 1060 | | | | until first 1061 | | | | Atomic Response 1062 | | | | has been 1063 | | | | generated 1064 ----------+------------+-------------+-------------+------------------- 1065 Send | Atomic | No Placement| Atomic | Atomic Response 1066 | | Guarantee | Response | Message will not 1067 | | between Send| will not be | be generated until 1068 | | Payload and | Placed at | Send has been 1069 | | Atomic | the Local | Completed 1070 | | Request | Peer Until | 1071 | | | Send Payload| 1072 | | | is Placed | 1073 | | | at the | 1074 | | | Remote Peer | 1075 ----------+------------+-------------+-------------+------------------- 1076 RDMA | Atomic | No Placement| Atomic | Not 1077 Write | | Guarantee | Response | Applicable 1078 | | between RDMA| will not be | 1079 | | Write | Placed at | 1080 | | Payload and | the Local | 1081 | | Atomic | Peer Until | 1082 | | Request | RDMA Write | 1083 | | | Payload | 1084 | | | is Placed | 1085 | | | at the | 1086 | | | Remote Peer | 1087 ----------+------------+-------------+-------------+------------------- 1088 RDMA | Atomic | No Placement| No Placement| Atomic Response 1089 Read | | Guarantee | Guarantee | Message will 1090 | | between | between | not be generated 1091 | | Atomic | Atomic | until RDMA 1092 | | Request and | Response | Read Response 1093 | | RDMA Read | and RDMA | has been 1094 | | Request | Read | generated 1095 | | | Response | 1096 ----------+------------+-------------+-------------+------------------- 1098 8. Error Processing 1100 In addition to error processing described in section 7 of [RFC5040], 1101 the following rules apply for the new RDMA Messages defined in this 1102 specification. 1104 8.1. Errors Detected at the Local Peer 1106 The Local Peer MUST send a Terminate Message for each of the 1107 following cases: 1109 1. For errors detected while creating an Atomic Request, Atomic 1110 Response, Immediate Data, or Immediate Data with SE Message, or 1111 other reasons not directly associated with an incoming Message, 1112 the Terminate Message and Error code are sent instead of the 1113 Message. In this case, the Error Type and Error Code fields are 1114 included in the Terminate Message, but the Terminated DDP Header 1115 and Terminated RDMA Header fields are set to zero. 1117 2. For errors detected on an incoming Atomic Request, Atomic 1118 Response, Immediate Data, or Immediate Data with Solicited Event 1119 (after the Message has been Delivered by DDP), the Terminate 1120 Message is sent at the earliest possible opportunity, preferably 1121 in the next outgoing RDMA Message. In this case, the Error Type, 1122 Error Code, and Terminated DDP Header fields are included in the 1123 Terminate Message, but the Terminated RDMA Header field is set to 1124 zero. 1126 8.2. Errors Detected at the Remote Peer 1128 On incoming Atomic Requests, Atomic Responses, Immediate Data, and 1129 Immediate Data with Solicited Event, the following MUST be 1130 validated: 1132 . The DDP layer MUST validate all DDP Segment fields. 1134 . The RDMA OpCode MUST be valid. 1136 . The RDMA Version MUST be valid. 1138 On incoming Atomic requests the following additional validation MUST 1139 be performed: 1141 . The RDMAP layer MUST validate that the Remote Peer's Tagged ULP 1142 Buffer address references a 64-bit aligned ULP Buffer address. In 1143 the case of an error, the RDMAP layer MUST generate a Terminate 1144 Message indicating RDMA Layer Remote Operation Error with Error 1145 Code Name "Catastrophic Error, Localized to RDMAP Stream" as 1146 described in Section 4.8 of [RFC5040]. Implementation Note: A ULP 1147 implementation can avoid this error by having the target ULP 1148 buffer of an atomic operation 64-bit aligned. 1150 9. Security Considerations 1152 This document specifies extensions to the RDMA Protocol 1153 specification in [RFC5040], and as such the Security Considerations 1154 discussed in Section 8 of [RFC5040] apply. No additional Security 1155 Considerations are required for the extensions specified in this 1156 document. 1158 10. IANA Considerations 1160 IANA is requested to add the following entries to the "RDMAP Message 1161 Operation Codes" registry of "RDDP Registries": 1163 0x8, Immediate Data, [RFCXXXX] 1165 0x9, Immediate Data with Solicited Event, [RFCXXXX] 1167 0xA, Atomic Request, [RFCXXXX] 1169 0xB, Atomic Response, [RFCXXXX] 1171 In addition, the following registry is requested to be added to 1172 "RDDP Registries". The following section specifies the registry, its 1173 initial contents and the administration policy in more detail. 1175 10.1. RDMAP Message Atomic Operation Subcodes 1177 Name of the registry: "RDMAP Message Atomic Operation Subcodes" 1179 Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit 1180 values [RFCXXXX]. 1182 Information that must be provided to assign a new value: An IESG- 1183 approved standards-track specification defining the semantics and 1184 interoperability requirements of the proposed new value and the 1185 fields to be recorded in the registry. 1187 Fields to record in the registry: RDMAP Message Atomic Operation 1188 Subcode, Atomic Operation, RFC Reference. 1190 Initial registry contents: 1192 0x0, FetchAdd, [RFCXXXX] 1194 0x1, Swap, [RFCXXXX] 1196 0x2, CmpSwap, [RFCXXXX] 1198 Note: An experimental RDMAP Message Operation Code has already been 1199 allocated; hence there is no need for an experimental RDMAP Message 1200 Atomic Operation Subcode. 1202 All other values are Unassigned and available to IANA for 1203 assignment. 1205 Allocation Policy: Standards Action ([RFC5226]) 1207 RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above 1208 with the RFC number of this document and remove this note. 1210 10.2. RDMAP Queue Numbers 1212 Name of the registry: "RDMAP DDP Untagged Queue Numbers" 1214 Namespace details: RDMAP DDP Untagged Queue numbers are 32-bit 1215 values [RFCXXXX]. 1217 Information that must be provided to assign a new value: An IESG- 1218 approved standards-track specification defining the semantics and 1219 interoperability requirements of the proposed new value and the 1220 fields to be recorded in the registry. 1222 Fields to record in the registry: RDMAP DDP Untagged Queue Numbers, 1223 Atomic Operation, RFC Reference. 1225 Initial registry contents: 1227 0x00000000, Queue 0 (Send operation Variants), [RFC5040] 1229 0x00000001, Queue 1 (RDMA Read Request operations), [RFC5040] 1231 0x00000002, Queue 2 (Terminate operations), [RFC5040] 1232 0x00000003, Queue 3 (Atomic Response operations), [RFCXXXX] 1234 Note: An experimental RDMAP Message Operation Code has already been 1235 allocated; hence there is no need for an experimental RDMAP DDP 1236 Untagged Queue Number. 1238 All other values are Unassigned and available to IANA for 1239 assignment. 1241 Allocation Policy: Standards Action ([RFC5226]) 1243 RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above 1244 with the RFC number of this document and remove this note. 1246 11. References 1248 11.1. Normative References 1250 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1251 Requirement Levels", BCP 14, RFC 2119, March 1997. 1253 [RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol 1254 Specification", RFC 5040, October 2007. 1256 [RFC5041] Shah, H. et al., "Direct Data Placement over Reliable 1257 Transports", RFC 5041, October 2007. 1259 [RFC5042] Pinkerton, J. and E. Deleganes, "Direct Data Placement 1260 Protocol (DDP) / Remote Direct Memory Access Protocol 1261 (RDMAP) Security", October 2007. 1263 11.2. Informative References 1265 [IB] Infiniband Trade Association, "Infiniband Architecture 1266 Specification Volumes 1 and 2", Release 1.1, November 1267 2002, available from http://www.infinibandta.org/specs. 1269 [RSOCKETS] RSockets, RDMA enabled Sockets library for Open Fabrics, 1270 available from 1271 http://git.openfabrics.org/git?p=~shefty/librdmacm.git;a=b 1272 lob;f=src/rsocket.c;h=d544dd097cda228de114173c8fe569dc1881 1273 f057;hb=HEAD. 1275 [DAT_ATOMICS] DAT Collaborative, User Direct Access Programming 1276 Library, "Ratified DAT IB extension spec", available from 1277 http://www.datcollaborative.org/DAT_IB_Extensions.pdf. 1279 12. Acknowledgments 1281 The authors would like to acknowledge the following contributors who 1282 provided valuable comments and suggestions. 1284 o David Black 1286 o Arkady Kanevsky 1288 o Bernard Metzler 1290 o Jim Pinkerton 1292 o Tom Talpey 1294 o Steve Wise 1296 o Don Wood 1298 This document was prepared using 2-Word-v2.0.template.dot. 1300 Appendix A. DDP Segment Formats for RDMA Messages 1302 This appendix is for information only and is NOT part of the 1303 standard. It simply depicts the DDP Segment format for the various 1304 RDMA Messages. 1306 A.1. DDP Segment for Atomic Operation Request 1308 The following figure depicts an Atomic Operation Request, DDP 1309 Segment: 1311 0 1 2 3 1312 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1313 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1314 | DDP Control | RDMA Control | 1315 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1316 | Reserved (Not Used) | 1317 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1318 | DDP (Atomic Operation Request) Queue Number | 1319 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1320 | DDP (Atomic Operation Request) Message Sequence Number | 1321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1322 | DDP (Atomic Operation Request) Message Offset | 1323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1324 | Reserved (Not Used) |AOpCode| 1325 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1326 | Request Identifier | 1327 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1328 | Remote STag | 1329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1330 | Remote Tagged Offset | 1331 + + 1332 | | 1333 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1334 | Add or Swap Data | 1335 + + 1336 | | 1337 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1338 | Add or Swap Mask | 1339 + + 1340 | | 1341 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1342 | Compare Data | 1343 + + 1344 | | 1345 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1346 | Compare Mask | 1347 + + 1348 | | 1349 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1351 A.2. DDP Segment for Atomic Response 1353 The following figure depicts an Atomic Operation Response, DDP 1354 Segment: 1356 0 1 2 3 1357 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1358 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1359 | DDP Control | RDMA Control | 1360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1361 | Reserved (Not Used) | 1362 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1363 | DDP (Atomic Operation Request) Queue Number | 1364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1365 | DDP (Atomic Operation Request) Message Sequence Number | 1366 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1367 | DDP (Atomic Operation Request) Message Offset | 1368 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1369 | Original Request Identifier | 1370 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1371 | Original Remote Value | 1372 + + 1373 | | 1374 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1376 A.3. DDP Segment for Immediate Data and Immediate Data with SE 1378 The following figure depicts an Immediate Data or Immediate data 1379 with SE, DDP Segment: 1381 0 1 2 3 1382 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1383 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1384 | DDP Control | RDMA Control | 1385 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1386 | Reserved (Not Used) | 1387 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1388 | DDP (Send) Queue Number | 1389 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1390 | DDP (Send) Message Sequence Number | 1391 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1392 | DDP Message Offset | 1393 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1394 | Immediate Data | 1395 + + 1396 | | 1397 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1399 Authors' Addresses 1401 Hemal Shah 1402 Broadcom Corporation 1403 5300 California Avenue 1404 Irvine, CA 92617 1405 Phone: 1-949-926-6941 1406 Email: hemal@broadcom.com 1408 Felix Marti 1409 Chelsio Communications, Inc. 1410 370 San Aleso Ave. 1411 Sunnyvale, CA 94085 1412 Phone: 1-408-962-3600 1413 Email: felix@chelsio.com 1415 Asgeir Eiriksson 1416 Chelsio Communications, Inc. 1417 370 San Aleso Ave. 1418 Sunnyvale, CA 94085 1419 Phone: 1-408-962-3600 1420 Email: asgeir@chelsio.com 1422 Wael Noureddine 1423 Chelsio Communications, Inc. 1424 370 San Aleso Ave. 1425 Sunnyvale, CA 94085 1426 Phone: 1-408-962-3600 1427 Email: wael@chelsio.com 1429 Robert Sharp 1430 Intel Corporation 1431 1300 South Mopac Expy, Mailstop: AN4-4B 1432 Austin, TX 78746 1433 Phone: 1-512-362-1407 1434 Email: robert.o.sharp@intel.com