idnits 2.17.1 draft-ietf-storm-rdmap-ext-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC5040]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 767 has weird spacing: '...e Data and ...' -- The document date (January 9, 2013) is 4096 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFCXXXX' is mentioned on line 1079, but not defined ** Obsolete normative reference: RFC 5226 (Obsoleted by RFC 8126) Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Storage Maintenance (storm) Working Group Hemal Shah 2 Internet Draft Broadcom Corporation 3 Intended status: Standards Track Felix Marti 4 Expires: July 2013 Wael Noureddine 5 Asgeir Eiriksson 6 Chelsio Communications, Inc. 7 Robert Sharp 8 Intel Corporation 9 January 9, 2013 11 RDMA Protocol Extensions 12 draft-ietf-storm-rdmap-ext-04.txt 14 Status of this Memo 16 This Internet-Draft is submitted to IETF in full conformance with 17 the provisions of BCP 78 and BCP 79. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF). Note that other groups may also distribute 21 working documents as Internet-Drafts. The list of current Internet- 22 Drafts is at http://datatracker.ietf.org/drafts/current. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as "work in progress." 29 This Internet-Draft will expire on July 9, 2013. 31 Copyright Notice 33 Copyright (c) 2013 IETF Trust and the persons identified as the 34 document authors. All rights reserved. 36 This document is subject to BCP 78 and the IETF Trust's Legal 37 Provisions Relating to IETF Documents 38 (http://trustee.ietf.org/license-info) in effect on the date of 39 publication of this document. Please review these documents 40 carefully, as they describe your rights and restrictions with 41 respect to this document. Code Components extracted from this 42 document must include Simplified BSD License text as described in 43 Section 4.e of the Trust Legal Provisions and are provided without 44 warranty as described in the Simplified BSD License. 46 Abstract 48 This document specifies extensions to the IETF Remote Direct Memory 49 Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write 50 services directly to applications and enables data to be transferred 51 directly into Upper Layer Protocol (ULP) Buffers without 52 intermediate data copies. The extensions specified in this document 53 provide the following capabilities and/or improvements: Atomic 54 Operations and Immediate Data. 56 Table of Contents 58 1. Introduction...................................................3 59 2. Requirements Language..........................................3 60 3. Glossary.......................................................3 61 4. Header Format Extensions.......................................5 62 4.1. RDMAP Control and Invalidate STag Fields..................5 63 4.2. RDMA Message Definitions..................................6 64 5. Atomic Operations..............................................7 65 5.1. Atomic Operation Details..................................8 66 5.1.1. FetchAdd.............................................8 67 5.1.2. Swap.................................................9 68 5.1.3. CmpSwap.............................................10 69 5.2. Atomic Operations........................................11 70 5.2.1. Atomic Operation Request Message....................12 71 5.2.2. Atomic Operation Response Message...................15 72 5.3. Atomicity Guarantees.....................................17 73 5.4. Atomic Operations Ordering and Completion Rules..........17 74 6. Immediate Data................................................18 75 6.1. RDMAP Interactions with ULP for Immediate Data...........18 76 6.2. Immediate Data Header Format.............................18 77 6.3. Immediate Data or Immediate Data with SE Message.........19 78 6.4. Ordering and Completions.................................20 79 7. Ordering and Completions Table................................20 80 8. Error Processing..............................................23 81 8.1. Errors Detected at the Local Peer........................23 82 8.2. Errors Detected at the Remote Peer.......................24 84 9. Security Considerations.......................................24 85 10. IANA Considerations..........................................24 86 10.1. RDMAP Message Atomic Operation Subcodes.................25 87 11. References...................................................26 88 11.1. Normative References....................................26 89 11.2. Informative References..................................26 90 12. Acknowledgments..............................................26 91 Appendix A. DDP Segment Formats for RDMA Messages................27 92 A.1. DDP Segment for Atomic Operation Request.................27 93 A.2. DDP Segment for Atomic Response..........................29 94 A.3. DDP Segment for Immediate Data and Immediate Data with SE29 96 1. Introduction 98 The RDMA Protocol [RFC5040] provides capabilities for zero copy and 99 kernel bypass data communications. This document specifies the 100 following extensions to the RDMA Protocol (RDMAP): 102 o Atomic operations on remote memory locations. Support for atomic 103 operation enhances the usability of RDMAP in distributed shared 104 memory environments. 106 o Immediate Data messages allow the ULP at the sender to provide a 107 small amount of data following an RDMA Message. 109 Other RDMA transport protocols define the functionality added by 110 these extensions leading to differences in RDMA applications and/or 111 Upper Layer Protocols. Removing these differences in the transport 112 protocols simplifies these applications and ULPs and that is the 113 main motivation for the extensions specified in this document. 115 2. Requirements Language 117 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 118 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 119 document are to be interpreted as described in RFC-2119 [RFC2119]. 121 3. Glossary 123 This document is an extension of [RFC5040] and key words are defined 124 in the glossary of the referenced document. 126 Atomic Operation - is an operation that results in an execution of a 127 64-bit operation at a specific address on a remote node. The 128 consumer can use Atomic Operations to read, modify and write at the 129 destination address while at the same time guarantee that no other 130 read or write operation will occur across any other RDMAP Streams on 131 an RNIC at the Data Sink. 133 Atomic Operation Request - An RDMA Message used by the Data Source 134 to perform an Atomic Operation at the Data Sink. 136 Atomic Operation Response - An RDMA Message used by the Data Sink to 137 describe the completion of an Atomic Operation at the Data Sink. 139 CmpSwap - is an Atomic Operation that is used to compare and swap a 140 value at a specific address on a remote node. 142 FetchAdd - is an Atomic Operation that is used to atomically 143 increment a value at a specific address on a remote node. 145 Immediate Data - a small fixed size portion of data sent from the 146 Data Source to a Data Sink 148 Immediate Data Message - An RDMA Message used by the Data Source to 149 send Immediate Data to the Data Sink 151 Immediate Data with Solicited Event (SE) Message - An RDMA Message 152 used by the Data Source to send Immediate Data with Solicited Event 153 to the Data Sink 155 Requester - the sender of an RDMA Atomic Operation request. 157 Responder - the receiver of an RDMA Atomic Operation request. 159 Swap - is an Atomic Operation that is used to swap a value at a 160 specific address on a remote node. 162 4. Header Format Extensions 164 The control information of RDMA Messages is included in DDP protocol 165 [RFC5041] defined header fields, with the following new formats: 166 . Four new RDMA Messages carry additional RDMAP headers. The 167 Immediate Data operation and Immediate Data with Solicited Event 168 operation include 8 bytes of data following the RDMAP header. 169 Atomic Operations include Atomic Request or Atomic Response 170 headers following the RDMAP header. 172 4.1. RDMAP Control and Invalidate STag Fields 174 The RDMA Messages defined by this specification use all 8 bits of 175 the RDMAP Control Field. The first octet reserved for ULP use in the 176 DDP Protocol MUST be used by the RDMAP to carry the RDMAP Control 177 Field. The ordering of the bits in the first octet MUST be as shown 178 in Figure 1. 180 Figure 1 depicts the format of the DDP Control and RDMAP Control 181 fields, in the style and convention of [RFC5040]: 183 0 1 2 3 184 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 186 |T|L| Resrv | DV| RV|Rsv| Opcode| 187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 188 | Invalidate STag | 189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 191 Figure 1 DDP Control and RDMAP Control Fields 193 Figure 2 defines the values of RDMA Opcode field that MUST be used 194 for the RDMA Messages defined in this specification. 196 Figure 2 also defines when the STag, Tagged Offset, and Queue Number 197 fields MUST be provided for the RDMA Messages defined in this 198 specification. 200 All RDMA Messages defined in this specification MUST have: 202 The RDMA Version (RV) field: 01b. 204 Opcode field: See Figure 2. 206 Invalidate STag: MUST be set to zero by the sender, ignored by the 207 receiver. 209 -------+-----------+-------+------+-------+-----------+-------------- 210 RDMA | Message | Tagged| STag | Queue | Invalidate| Message 211 Opcode | Type | Flag | and | Number| STag | Length 212 | | | TO | | | Communicated 213 | | | | | | between DDP 214 | | | | | | and RDMAP 215 -------+-----------+-------+------+-------+-----------+-------------- 216 1000b | Immediate | 0 | N/A | 0 | N/A | Yes 217 | Data | | | | | 218 -------+-----------+------------------------------------------------- 219 1001b | Immediate | 0 | N/A | 0 | N/A | Yes 220 | Data with | | | | | 221 | SE | | | | | 222 -------+-----------+------------------------------------------------- 223 1010b | Atomic | 0 | N/A | 1 | N/A | Yes 224 | Request | | | | | 225 -------+-----------+------------------------------------------------- 226 1011b | Atomic | 0 | N/A | 3 | N/A | Yes 227 | Response | | | | | 228 -------+-----------+------------------------------------------------- 230 Figure 2 Additional RDMA Usage of DDP Fields 232 Note: N/A means Not Applicable. 234 All other DDP and RDMAP control fields MUST be set as described in 235 [RFC5040]. 237 4.2. RDMA Message Definitions 239 The following figure defines which RDMA Headers MUST be used on each 240 new RDMA Message and which new RDMA Messages are allowed to carry 241 ULP payload: 243 -------+-----------+-------------------+------------------------- 244 RDMA | Message | RDMA Header Used | ULP Message allowed in 245 Message| Type | | the RDMA Message 246 OpCode | | | 247 | | | 248 -------+-----------+-------------------+------------------------- 249 1000b | Immediate | Immediate Data | No 250 | Data | Header | 251 -------+-----------+-------------------+------------------------- 252 1001b | Immediate | Immediate Data | No 253 | Data with | Header | 254 | SE | | 255 -------+-----------+-------------------+------------------------- 256 1010b | Atomic | Atomic Request | No 257 | Request | Header | 258 -------+-----------+-------------------+------------------------- 259 1011b | Atomic | Atomic Response | No 260 | Response | Header | 261 -------+-----------+-------------------+------------------------- 262 Figure 3 RDMA Message Definitions 264 5. Atomic Operations 266 The RDMA Protocol Specification in [RFC5040] does not include 267 support for Atomic Operations which are an important building block 268 for implementing distributed shared memory. 270 This document extends the RDMA Protocol specification with a set of 271 basic Atomic Operations, and specifies their resource and ordering 272 rules. 274 Atomic operations as specified in this document execute a 64-bit 275 operation at a specified destination address on a remote node. The 276 operations atomically read, modify and write back the contents of 277 the destination address and guarantee that Atomic Operations on this 278 address by other RDMAP Streams on the same RNIC do not occur between 279 the read and the write. Atomic Operations as specified in this 280 document MAY be implemented. The discovery of whether the Atomic 281 Operations are implemented or not is outside the scope of this 282 specification and it should be handled by the ULPs or applications. 283 The advertisement of Tagged Buffer information for Atomic Operations 284 is outside the scope of this specification and it must be handled by 285 the ULPs. 287 Implementation note: It is recommended that the applications do not 288 use the buffer addresses used for Atomic Operations for other RDMA 289 operations. 291 Atomic Operations use the same remote addressing mechanism as RDMA 292 Reads and Writes. The buffer address specified in the request is in 293 the address space of the Remote Peer that the Atomic Operation is 294 targeted at. 296 5.1. Atomic Operation Details 298 The following sub-sections describe the Atomic Operations in more 299 details. 301 5.1.1. FetchAdd 303 The FetchAdd Atomic Operation requests the Responder to read a 64- 304 bit Original Remote Data value at a 64-bit aligned buffer address in 305 the Responder's memory, to perform FetchAdd operation on multiple 306 fields of selectable length specified by 64-bit "Add Mask", and 307 write the result back to the same virtual address. The Atomic 308 addition is performed independently on each one of these fields. A 309 bit set in the Add Mask field specifies the field boundary. FetchAdd 310 Atomic Operations MUST target buffer addresses that are 64-bit 311 aligned. FetchAdd Atomic Operations that target buffer addresses 312 that are not 64-bit aligned MUST be surfaced as errors and the 313 Responder's memory MUST NOT modified in such cases. Additionally an 314 error MUST be surfaced and a terminate message MUST be generated. 315 The setting of "Add Mask" field to 0x0000000000000000 results in 316 Atomic Add of 64-bit Original Remote Data Value and 64-bit "Add 317 Data". 319 The pseudo code below describes masked FetchAdd Atomic Operation. 321 bit_location = 1 323 carry = 0 325 Remote Data Value = 0 327 for bit = 0 to 63 329 { 331 if (bit != 0 ) bit_location = bit_location << 1 332 val1 = !(!(Original Remote Data Value & bit_location)) 334 val2 = !(!(Add Data & bit_location)) 336 sum = carry + val1 + val2 338 carry = !(!(sum & 2)) 340 sum = sum & 1 342 if (sum) 344 Remote Data Value |= bit_location 346 carry = ((carry) && (!(Add Mask & bit_location))) 348 } 350 The FetchAdd operation is performed in the endian format of the 351 target memory. The "Original Remote Data" is converted from the 352 endian format of the target memory for return and returned to the 353 Requester. The fields are in big-endian format on the wire. 355 The Requester specifies: 357 o Remote STag 359 o Remote Tagged Offset 361 o Add Data 363 o Add Mask 365 The Responder returns: 367 o Original Remote Data 369 5.1.2. Swap 371 The Swap Atomic Operation requires the Responder to read a 64-bit 372 value at a 64-bit aligned buffer address in the Responder's memory, 373 then to write the "Swap Data" fields into the same buffer address. 374 The "Original Remote Data" is converted from the endian format of 375 the target memory for return and returned to the Requester. The 376 fields are in big-endian format on the wire. 378 The Requester specifies: 380 o Remote STag 382 o Remote Tagged Offset 384 o Swap Data 386 The Responder returns: 388 o Original Remote Data 390 After the successful completion of Swap operation, the Responder's 391 memory at the specified buffer address MUST contain the "Swap Data" 392 field in the header. Swap Atomic Operations MUST target buffer 393 addresses that are 64-bit aligned. Swap Atomic Operations that 394 target buffer addresses that are not 64-bit aligned MUST be surfaced 395 as errors and the Responder's memory MUST NOT be modified in such 396 cases. Additionally an error MUST be surfaced and a terminate 397 message MUST be generated. 399 5.1.3. CmpSwap 401 The CmpSwap Atomic Operation requires the Responder to read a 64-bit 402 value at a 64-bit aligned buffer address in the Responder's memory, 403 to perform an AND logical operation using the 64 bit "Compare Mask" 404 field in the Atomic Operation Request header, then to compare it 405 with the result of a logical AND operation of the "Compare Mask" and 406 the "Compare Data" fields in the header, and, if the two values are 407 equal, to swap masked bits in the same buffer address with the 408 masked Swap Data. If the two masked compare values are not equal, 409 the contents of the Responder's memory are not changed. In either 410 case, the original value read from the buffer address is converted 411 from the endian format of the target memory for return and returned 412 to the Requester. The fields are in big-endian format on the wire. 414 The Requester specifies: 416 o Remote STag 418 o Remote Tagged Offset 420 o Swap Data 422 o Swap Mask 423 o Compare Data 425 o Compare Mask 427 The Responder returns: 429 o Original Remote Data Value 431 The following pseudo code describes the masked CmpSwap operation 432 result. 434 if (!((Compare Data ^ Original Remote Data value) & Compare Mask) 436 then 438 Remote Data Value = 440 (Original Remote Data Value & ~(Swap Mask)) 442 | (Swap Data & Swap Mask) 444 else 446 Remote Data Value = Original Remote Data Value 448 After the operation, the remote data buffer MUST contain the 449 "Original Remote Data Value" (if comparison did not match) or the 450 masked "Swap Data" (if the comparison did match). CmpSwap Atomic 451 Operations MUST target buffer addresses that are 64-bit aligned. 452 CmpSwap Atomic Operations that target buffer addresses that are not 453 64-bit aligned MUST be surfaced as errors and the remote data buffer 454 MUST NOT be modified in such cases. Additionally an error MUST be 455 surfaced and a terminate message MUST be generated. 457 5.2. Atomic Operations 459 The Atomic Operation Request and Response are RDMA Messages. An 460 Atomic Operation makes use of the DDP Untagged Buffer Model. Atomic 461 Operations use the same Queue Number as RDMA Read Requests (QN=1). 462 Reusing the same Queue Number allows the Atomic Operations to reuse 463 the same infrastructure (e.g. ORD/IRD flow control) as defined for 464 RDMA Read Requests. 466 The RDMA Message OpCode for an Atomic Request Message is 1010b. The 467 RDMA Message OpCode for an Atomic Response Message is 1011b. 469 5.2.1. Atomic Operation Request Message 471 The Atomic Operation Request Message carries an Atomic Operation 472 Header that describes the buffer address in the Responder's memory. 473 The Atomic Operation Request header immediately follows the DDP 474 header. The RDMAP layer passes to the DDP layer a RDMAP Control 475 Field. The following figure depicts the Atomic Operation Request 476 Header that MUST be used for all Atomic Operation Request Messages: 478 0 1 2 3 479 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | Reserved (Not Used) |AOpCode| 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 483 | Request Identifier | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 485 | Remote STag | 486 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 487 | Remote Tagged Offset | 488 + + 489 | | 490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 491 | Add or Swap Data | 492 + + 493 | | 494 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 495 | Add or Swap Mask | 496 + + 497 | | 498 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 499 | Compare Data | 500 + + 501 | | 502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 | Compare Mask | 504 + + 505 | | 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 508 Figure 4 Atomic Operation Request Header 510 Reserved (Not Used): 28 bits 511 This field MUST be set to zero on transmit, ignored on 512 receive. 514 Atomic Operation Code (AOpCode): 4 bits. 516 See Figure 5. 518 Request Identifier: 32 bits. 520 The Request Identifier specifies a number that is used to 521 identify Atomic Operation Request Message. The use of this 522 field is implementation dependent and outside the scope of 523 this specification. 525 Remote STag: 32 bits. 527 The Remote STag identifies the Remote Peer's Tagged Buffer 528 targeted by the Atomic Operation. The Remote STag is 529 associated with the RDMAP Stream through a mechanism that is 530 outside the scope of the RDMAP specification. 532 Remote Tagged Offset: 64 bits. 534 The Remote Tagged Offset specifies the starting offset, in 535 octets, from the base of the Remote Peer's Tagged Buffer 536 targeted by the Atomic Operation. The Remote Tagged Offset MAY 537 start at an arbitrary offset. 539 Add or Swap Data: 64 bits. 541 The Add or Swap Data field specifies the 64-bit "Add Data" 542 value in an Atomic FetchAdd Operation or the 64-bit "Swap 543 Data" value in an Atomic Swap or CmpSwap Operation. 545 Add or Swap Mask: 64 bits 547 This field is used in masked Atomic Operations (FetchAdd and 548 CmpSwap) to perform a bitwise logical AND operation as specified 549 in the definition of these operations. For non-masked Atomic 550 Operations (Swap), this field MUST be set to ffffffffffffffffh on 551 transmit and ignored by the receiver. 553 Compare Data: 64 bits. 555 The Compare Data field specifies the 64-bit "Compare Data" 556 value in an Atomic CmpSwap Operation. For Atomic FetchAdd and 557 Atomic Swap operation, the Compare Data field MUST be set to 558 zero on transmit and ignored by the receiver. 560 Compare Mask: 64 bits 562 This field is used in masked Atomic Operation CmpSwap to 563 perform a bitwise logical AND operation as specified in the 564 definition of these operations. For Atomic Operations FetchAdd 565 and Swap, this field MUST be set to ffffffffffffffffh on 566 transmit and ignored by the receiver. 568 ---------+-----------+----------+----------+---------+--------- 569 Atomic | Atomic | Add or | Add or | Compare | Compare 570 Operation| Operation | Swap | Swap | Data | Mask 571 Code | | Data | Mask | | 572 ---------+-----------+----------+----------+---------+--------- 573 0000b | FetchAdd | Add Data | Add Mask | N/A | N/A 574 ---------+-----------+----------+----------+---------+--------- 575 0001b | Swap | Swap Data| N/A | N/A | N/A 576 ---------+-----------+----------+----------+---------+--------- 577 0010b | CmpSwap | Swap Data| Swap Mask| Valid | Valid 578 ---------+-----------+----------+----------+---------+--------- 579 0011b | | 580 to | Reserved | Not Specified 581 1111b | | 582 ---------+-----------+----------------------------------------- 584 Figure 5 Atomic Operation Message Definitions 586 The Atomic Operation Request Message has the following semantics: 588 1. An Atomic Operation Request Message MUST reference an Untagged 589 Buffer. That is, the Local Peer's RDMAP layer MUST request that 590 the DDP mark the Message as Untagged. 592 2. One Atomic Operation Request Message MUST consume one Untagged 593 Buffer. 595 3. The Responder's RDMAP layer MUST process an Atomic Operation 596 Request Message. A valid Atomic Operation Request Message MUST 597 NOT be delivered to the Responder's ULP (i.e., it is processed by 598 the RDMAP layer). 600 4. At the Responder, when an invalid Atomic Operation Request 601 Message is delivered to the Remote Peer's RDMAP layer, an error 602 is surfaced. 604 5. An Atomic Operation Request Message MUST reference the RDMA Read 605 Request Queue. That is, the Requester's RDMAP layer MUST request 606 that the DDP layer set the Queue Number field to one. 608 6. The Requester MUST pass to the DDP layer Atomic Operation Request 609 Messages in the order they were submitted by the ULP. 611 7. The Responder MUST process the Atomic Operation Request Messages 612 in the order they were sent. 614 8. If the Responder receives a valid Atomic Operation Request 615 Message, it MUST respond with a valid Atomic Operation Response 616 Message. 618 5.2.2. Atomic Operation Response Message 620 The Atomic Operation Response Message carries an Atomic Operation 621 Response Header that contains the "Original Request Identifier" and 622 "Original Remote Data Value". The Atomic Operation Response Header 623 immediately follows the DDP header. The RDMAP layer passes to the 624 DDP layer a RDMAP Control Field. The following figure depicts the 625 Atomic Operation Response header that MUST be used for all Atomic 626 Operation Response Messages: 628 0 1 2 3 629 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 630 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 631 | Original Request Identifier | 632 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 633 | Original Remote Data Value | 634 + + 635 | | 636 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 638 Figure 6 Atomic Operation Response Header 640 Original Request Identifier: 32 bits. 642 The Original Request Identifier MUST be set to the value 643 specified in the Request Identifier field that was originally 644 provided in the corresponding Atomic Operation Request 645 Message. 647 Original Remote Data Value: 64 bits. 649 The Original Remote Value specifies the original 64-bit value 650 stored at the buffer address targeted by the Atomic Operation. 652 The Atomic Operation Response Message has the following semantics: 654 1. The Atomic Operation Response Message for the associated Atomic 655 Operation Request Message travels in the opposite direction. 657 2. An Atomic Operation Response Message MUST consume an Untagged 658 Buffer. That is, the Responder RDMAP layer MUST request that the 659 DDP mark the Message as Untagged. 661 3. An Atomic Operation Response Message MUST reference the Queue 662 Number 3. That is, the Responder's RDMAP layer MUST request that 663 the DDP layer set the Queue Number field to 3. 665 4. The Responder MUST ensure that a sufficient number of Untagged 666 Buffers are available on the RDMA Read Request Queue (Queue with 667 DDP Queue Number 1) to support the maximum number of Atomic 668 Operation Requests negotiated by the ULP. 670 5. The RDMAP layer MUST Deliver the Atomic Operation Response 671 Message to the ULP. 673 6. At the Requester, when an invalid Atomic Operation Response 674 Message is delivered to the Remote Peer's RDMAP layer, an error 675 is surfaced. 677 7. The Responder RDMAP layer MUST pass Atomic Operation Response 678 Messages to the DDP layer, in the order that the Atomic Operation 679 Request Messages were received by the RDMAP layer, at the 680 Responder. 682 5.3. Atomicity Guarantees 684 Atomicity of the Read-Modify-Write (RMW) on the Responder's node by 685 the Atomic Operation MUST be assured in the presence of concurrent 686 atomic accesses by other RDMAP Streams on the same RNIC. 688 5.4. Atomic Operations Ordering and Completion Rules 690 In addition to the ordering and completion rules described in 691 [RFC5040], the following rules apply to implementations of the 692 Atomic operations. 694 1. For an Atomic operation, the contents of the Tagged Buffer at the 695 Responder MAY be indeterminate until the Atomic Operation 696 Response Message has been Delivered at the Requester. 698 2. Atomic Operation Request Messages MUST NOT start processing at 699 the Responder until they have been Delivered to RDMAP by DDP. 701 3. Atomic Operation Response Messages MAY be generated at the 702 Responder after subsequent RDMA Write Messages or Send Messages 703 have been Placed or Delivered. 705 4. Atomic Operation Response Message processing at the Responder 706 MUST be started only after the Atomic Operation Request Message 707 has been Delivered by the DDP layer (thus, all previous RDMA 708 Messages have been properly submitted for ordered Placement). 710 5. Send Messages MAY be Completed at the Responder before prior 711 incoming Atomic Operation Request Messages have completed their 712 response processing. 714 6. An Atomic Operation MUST NOT be Completed at the Requester until 715 the DDP layer Delivers the associated incoming Atomic Operation 716 Response Message. 718 7. If more than one outstanding Atomic Request Messages are 719 supported by both peers, the Atomic Operation Request Messages 720 MUST be processed in the order they were delivered by the DDP 721 layer on the Responder. Atomic Operation Response Messages MUST 722 be submitted to the DDP layer on the Responder in the order the 723 Atomic Operation Request Messages were Delivered by DDP. 725 6. Immediate Data 727 The Immediate Data operation is used in conjunction with an RDMA 728 Operation to improve ULP processing efficiency by allowing 8 bytes 729 of immediate data to be delivered with the completion of the 730 previous operation after the previous operation has been delivered 731 at the Remote Peer. 733 6.1. RDMAP Interactions with ULP for Immediate Data 735 For Immediate Data operations, the following are the interactions 736 between the RDMAP Layer and the ULP: 737 . At the Data Source: 739 . The ULP passes to the RDMAP Layer the following: 741 . Eight bytes of ULP Immediate Data 743 . When the Immediate Data operation Completes, an indication 744 of the Completion results. 746 . At the Data Sink: 748 . If the Immediate Data operation is Completed successfully, 749 the RDMAP Layer passes the following information to the ULP 750 Layer: 752 . Eight bytes of Immediate Data 754 . An Event, if the Data Sink is configured to generate an 755 Event and the RDMA Message Opcode indicates Message Type 756 Immediate Data with Solicited Event. 758 . If the Immediate Data operation is Completed in error, the 759 Data Sink RDMAP Layer will pass up the corresponding error 760 information to the Data Sink ULP and send a Terminate 761 Message to the Data Source RDMAP Layer. The Data Source 762 RDMAP Layer will then pass up the Terminate Message to the 763 ULP. 765 6.2. Immediate Data Header Format 767 The Immediate Data and Immediate Data with SE Messages carry 768 immediate data as shown in Figure 7. The RDMAP layer passes to the 769 DDP layer an RDMAP Control Field and 8 bytes of Immediate Data. The 770 first 8 bytes of the data following the DDP header contains the 771 Immediate Data. See section A.3. for the DDP segment format of an 772 Immediate Data or Immediate Data with SE Message. 774 0 1 2 3 775 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 776 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 777 | Immediate Data | 778 + + 779 | | 780 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 782 Figure 7 Immediate Data or Immediate Data with SE Message Header 784 Immediate Data: 64 bits. 785 Eight bytes of data transferred from the Requester to an 786 untagged buffer at the Responder. 788 6.3. Immediate Data or Immediate Data with SE Message 790 The Immediate Data or Immediate Data with SE Message uses the DDP 791 Untagged Buffer Model to transfer Immediate data from the Data 792 Source to the Data Sink. 793 . An Immediate Data or Immediate Data with SE Message MUST 794 reference an Untagged Buffer. That is, the Local Peer's RDMAP 795 Layer MUST request that the DDP layer mark the Message as 796 Untagged. 798 . One Immediate Data or Immediate Data with SE Message MUST consume 799 one Untagged Buffer. 801 . At the Remote Peer, the Immediate Data or Immediate Data with SE 802 Message MUST be Delivered to the Remote Peer's ULP in the order 803 they were sent. 805 . For an Immediate Data or Immediate Data with SE Message, the 806 Local Peer's RDMAP Layer MUST request that the DDP layer set the 807 Queue Number field to zero. 809 . For an Immediate Data or Immediate Data with SE Message, the 810 Local Peer's RDMAP Layer MUST request that the DDP layer transmit 811 8 bytes of data. 813 . The Local Peer MUST issue Immediate Data and Immediate Data with 814 SE Messages in the order they were submitted by the ULP. 816 . The Remote Peer MUST check that Immediate Data and Immediate Data 817 with SE Messages include exactly 8 bytes of data from the DDP 818 layer. 820 6.4. Ordering and Completions 822 Ordering and completion rules for Immediate Data are the same as 823 those for a Send operation as described in section 5.5 of RFC 5040. 825 7. Ordering and Completions Table 827 The following table summarizes the ordering relationships for Atomic 828 and Immediate Data operations from the standpoint of Local Peer issuing 829 the Operations. Note that in the table that follows, Send includes 830 Send, Send with Invalidate, Send with Solicited Event, and Send with 831 Solicited Event and Invalidate. Also note that in the table below, 832 Immediate Data includes Immediate Data and Immediate Data with 833 Solicited Event. 835 ----------+------------+-------------+-------------+------------------- 836 First | Second | Placement | Placement | Ordering 837 Operation | Operation | Guarantee at| Guarantee at| Guarantee at 838 | | Remote Peer | Local Peer | Remote Peer 839 ----------+------------+-------------+-------------+------------------- 840 Immediate | Send | No Placement| Not | Completed in 841 Data | | Guarantee | Applicable | Order 842 | | between Send| | 843 | | Payload and | | 844 | | Immediate | | 845 | | Data | | 846 ----------+------------+-------------+-------------+------------------- 847 Immediate | RDMA | No Placement| Not | Not 848 Data | Write | Guarantee | Applicable | Applicable 849 | | between RDMA| | 850 | | Write | | 851 | | Payload and | | 852 | | Immediate | | 853 | | Data | | 854 ----------+------------+-------------+-------------+------------------- 855 Immediate | RDMA | No Placement| RDMA Read | RDMA Read 856 Data | Read | Guarantee | Response | Response 857 | | between | will not be | Message will 858 | | Immediate | Placed until| not be 859 | | Data and | Immediate | generated 860 | | RDMA Read | Data is | until 861 | | Request | Placed at | Immediate Data 862 | | | Remote Peer | has been 863 | | | | Completed 864 ----------+------------+-------------+-------------+------------------- 865 Immediate | Atomic | No Placement| Atomic | Atomic 866 Data | | Guarantee | Response | Response 867 | | between | will not be | Message will 868 | | Immediate | Placed until| not be 869 | | Data and | Immediate | generated 870 | | Atomic | Data is | until 871 | | Request | Placed at | Immediate Data 872 | | | Remote Peer | has been 873 | | | | Completed 874 ----------+------------+-------------+-------------+------------------- 875 Immediate | Immediate | No Placement| Not | Completed in 876 Data or | Data | Guarantee | Applicable | Order 877 Send | | | | 878 ----------+------------+-------------+-------------+------------------- 879 RDMA Write| Immediate | No Placement| Not | Immediate Data 880 | Data | Guarantee | Applicable | is Completed 881 | | | | after RDMA 882 | | | | Write is Placed 883 | | | | and Delivered 884 ----------+------------+-------------+-------------+------------------- 885 RDMA Read | Immediate | No Placement| Immediate | Not Applicable 886 | Data | Guarantee | Data MAY be | 887 | | between | Placed | 888 | | Immediate | before | 889 | | Data and | RDMA Read | 890 | | RDMA Read | Response is | 891 | | Request | generated | 892 ----------+------------+-------------+-------------+------------------- 893 Atomic | Immediate | No Placement| Immediate | Not Applicable 894 | Data | Guarantee | Data MAY be | 895 | | between | Placed | 896 | | Immediate | before | 897 | | Data and | Atomic | 898 | | Atomic | Response is | 899 | | Request | generated | 900 ----------+------------+-------------+-------------+------------------- 901 Atomic | Send | No Placement| Send Payload| Not Applicable 902 | | Guarantee | MAY be | 903 | | between Send| Placed | 904 | | Payload and | before | 905 | | Atomic | Atomic | 906 | | Request | Response is | 907 | | | generated | 908 ----------+------------+-------------+-------------+------------------- 909 Atomic | RDMA | No Placement| RDMA Write | Not 910 | Write | Guarantee | Payload MAY | Applicable 911 | | between RDMA| be Placed | 912 | | Write | before | 913 | | Payload and | Atomic | 914 | | Atomic | Response is | 915 | | Request | generated | 916 ----------+------------+-------------+-------------+------------------- 917 Atomic | RDMA | No Placement| No Placement| RDMA Read 918 | Read | Guarantee | Guarantee | Response 919 | | between | between | Message will 920 | | Atomic | Atomic | not be 921 | | Request and | Response | generated 922 | | RDMA Read | and RDMA | until Atomic 923 | | Request | Read | Response Message 924 | | | Response | has been 925 | | | | generated 926 ----------+------------+-------------+-------------+------------------- 927 Atomic | Atomic | No Placement| No Placement| Second Atomic 928 | | Guarantee | Guarantee | Response 929 | | between two | between two | Message will 930 | | Atomic | Atomic | not be 931 | | Requests | Responses | generated 932 | | | | until first 933 | | | | Atomic Response 934 | | | | has been 935 | | | | generated 936 ----------+------------+-------------+-------------+------------------- 937 Send | Atomic | No Placement| Atomic | Atomic Response 938 | | Guarantee | Response | Message will not 939 | | between Send| will not be | be generated until 940 | | Payload and | Placed at | Send has been 941 | | Atomic | the Local | Completed 942 | | Request | Peer Until | 943 | | | Send Payload| 944 | | | is Placed | 945 | | | at the | 946 | | | Remote Peer | 947 ----------+------------+-------------+-------------+------------------- 948 RDMA | Atomic | No Placement| Atomic | Not 949 Write | | Guarantee | Response | Applicable 950 | | between RDMA| will not be | 951 | | Write | Placed at | 952 | | Payload and | the Local | 953 | | Atomic | Peer Until | 954 | | Request | Send Payload| 955 | | | is Placed | 956 | | | at the | 957 | | | Remote Peer | 958 ----------+------------+-------------+-------------+------------------- 959 RDMA | Atomic | No Placement| No Placement| Atomic Response 960 Read | | Guarantee | Guarantee | Message will 961 | | between | between | not be generated 962 | | Atomic | Atomic | until RDMA 963 | | Request and | Response | Read Response 964 | | RDMA Read | and RDMA | has been 965 | | Request | Read | generated 966 | | | Response | 967 ----------+------------+-------------+-------------+------------------- 969 8. Error Processing 971 In addition to error processing described in section 7 of [RFC5040], 972 the following rules apply for the new RDMA Messages defined in this 973 specification. 975 8.1. Errors Detected at the Local Peer 977 The Local Peer MUST send a Terminate Message for each of the 978 following cases: 980 1. For errors detected while creating an Atomic Request, Atomic 981 Response, Immediate Data, or Immediate Data with SE Message, or 982 other reasons not directly associated with an incoming Message, 983 the Terminate Message and Error code are sent instead of the 984 Message. In this case, the Error Type and Error Code fields are 985 included in the Terminate Message, but the Terminated DDP Header 986 and Terminated RDMA Header fields are set to zero. 988 2. For errors detected on an incoming Atomic Request, Atomic 989 Response, Immediate Data, or Immediate Data with Solicited Event 990 (after the Message has been Delivered by DDP), the Terminate 991 Message is sent at the earliest possible opportunity, preferably 992 in the next outgoing RDMA Message. In this case, the Error Type, 993 Error Code, and Terminated DDP Header fields are included in the 994 Terminate Message, but the Terminated RDMA Header field is set to 995 zero. 997 8.2. Errors Detected at the Remote Peer 999 On incoming Atomic Requests, Atomic Responses, Immediate Data, and 1000 Immediate Data with Solicited Event, the following MUST be 1001 validated: 1003 . The DDP layer MUST validate all DDP Segment fields. 1005 . The RDMA OpCode MUST be valid. 1007 . The RDMA Version MUST be valid. 1009 On incoming Atomic requests the following additional validation MUST 1010 be performed: 1012 . The RDMAP layer MUST validate that the Remote Peer's Tagged 1013 Buffer address references a 64-bit aligned ULP buffer address. In 1014 the case of an error, the RDMAP layer MUST generate a Terminate 1015 Message indicating RDMA Layer Remote Operation Error with Error 1016 Code Name "Catastrophic Error, Localized to RDMAP Stream" as 1017 described in Section 4.8 of [RFC5040]. Implementation Note: A ULP 1018 implementation can avoid this error by having the target ULP 1019 buffer of an atomic operation 64-bit aligned. 1021 9. Security Considerations 1023 This document specifies extensions to the RDMA Protocol 1024 specification in [RFC5040], and as such the Security Considerations 1025 discussed in Section 8 of [RFC5040] apply. 1027 10. IANA Considerations 1029 IANA is requested to add the following entries to the "RDMAP Message 1030 Operation Codes" registry of "RDDP Registries": 1032 0x8, Immediate Data, [RFCXXXX] 1034 0x9, Immediate Data with SE, [RFCXXXX] 1036 0xA, Atomic Request, [RFCXXXX] 1038 0xB, Atomic Response, [RFCXXXX] 1040 In addition, the following registry is requested to be added to 1041 "RDDP Registries". The following section specifies the registry, its 1042 initial contents and the administration policy in more detail. 1044 10.1. RDMAP Message Atomic Operation Subcodes 1046 Name of the registry: "RDMAP Message Atomic Operation Subcodes" 1048 Namespace details: RDMAP Message Atomic Operation Subcodes are 4-bit 1049 values [RFCXXXX]. 1051 Information that must be provided to assign a new value: An IESG- 1052 approved standards-track specification defining the semantics and 1053 interoperability requirements of the proposed new value and the 1054 fields to be recorded in the registry. 1056 Assignment policy: If the requested value is not already assigned, 1057 it may be assigned to the requester. 1059 Fields to record in the registry: RDMAP Message Atomic Operation 1060 Subcode, Atomic Operation, RFC Reference. 1062 Initial registry contents: 1064 0x0, FetchAdd, [RFCXXXX] 1066 0x1, Swap, [RFCXXXX] 1068 0x2, CmpSwap, [RFCXXXX] 1070 Note: An experimental RDMAP Message Operation Code has already been 1071 allocated; hence there is no need for an experimental RDMAP Message 1072 Atomic Operation Subcode. 1074 All other values are Unassigned and available to IANA for 1075 assignment. 1077 Allocation Policy: Standards Action ([RFC5226]) 1079 RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above 1080 with the RFC number of this document and remove this note. 1082 11. References 1084 11.1. Normative References 1086 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1087 Requirement Levels", BCP 14, RFC 2119, March 1997. 1089 [RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol 1090 Specification", RFC 5040, October 2007. 1092 [RFC5041] Shah, H. et al., "Direct Data Placement over Reliable 1093 Transports", RFC 5041, October 2007. 1095 [RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an 1096 IANA Considerations Section in RFCs", RFC 5226, BCP 26, 1097 May 2008. 1099 11.2. Informative References 1101 12. Acknowledgments 1103 The authors would like to acknowledge the following contributors who 1104 provided valuable comments and suggestions. 1106 o David Black 1108 o Arkady Kanevsky 1110 o Bernard Metzler 1112 o Jim Pinkerton 1114 o Tom Talpey 1116 o Steve Wise 1118 This document was prepared using 2-Word-v2.0.template.dot. 1120 Appendix A. DDP Segment Formats for RDMA Messages 1122 This appendix is for information only and is NOT part of the 1123 standard. It simply depicts the DDP Segment format for the various 1124 RDMA Messages. 1126 A.1. DDP Segment for Atomic Operation Request 1128 The following figure depicts an Atomic Operation Request, DDP 1129 Segment: 1131 0 1 2 3 1132 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1133 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1134 | DDP Control | RDMA Control | 1135 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1136 | Reserved (Not Used) | 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1138 | DDP (Atomic Operation Request) Queue Number | 1139 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1140 | DDP (Atomic Operation Request) Message Sequence Number | 1141 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1142 | DDP (Atomic Operation Request) Message Offset | 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1144 | Reserved (Not Used) |AOpCode| 1145 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1146 | Request Identifier | 1147 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1148 | Remote STag | 1149 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1150 | Remote Tagged Offset | 1151 + + 1152 | | 1153 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1154 | Add or Swap Data | 1155 + + 1156 | | 1157 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1158 | Add or Swap Mask | 1159 + + 1160 | | 1161 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1162 | Compare Data | 1163 + + 1164 | | 1165 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1166 | Compare Mask | 1167 + + 1168 | | 1169 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1171 A.2. DDP Segment for Atomic Response 1173 The following figure depicts an Atomic Operation Response, DDP 1174 Segment: 1176 0 1 2 3 1177 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1179 | DDP Control | RDMA Control | 1180 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1181 | Reserved (Not Used) | 1182 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1183 | DDP (Atomic Operation Request) Queue Number | 1184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1185 | DDP (Atomic Operation Request) Message Sequence Number | 1186 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1187 | DDP (Atomic Operation Request) Message Offset | 1188 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1189 | Original Request Identifier | 1190 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1191 | Original Remote Value | 1192 + + 1193 | | 1194 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1196 A.3. DDP Segment for Immediate Data and Immediate Data with SE 1198 The following figure depicts an Immediate Data or Immediate data 1199 with SE, DDP Segment: 1201 0 1 2 3 1202 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1203 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1204 | DDP Control | RDMA Control | 1205 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1206 | Reserved (Not Used) | 1207 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1208 | DDP (Send) Queue Number | 1209 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1210 | DDP (Send) Message Sequence Number | 1211 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1212 | DDP Message Offset | 1213 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1214 | Immediate Data | 1215 + + 1216 | | 1217 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1219 Authors' Addresses 1221 Hemal Shah 1222 Broadcom Corporation 1223 5300 California Avenue 1224 Irvine, CA 92617 1225 Phone: 1-949-926-6941 1226 Email: hemal@broadcom.com 1228 Felix Marti 1229 Chelsio Communications, Inc. 1230 370 San Aleso Ave. 1231 Sunnyvale, CA 94085 1232 Phone: 1-408-962-3600 1233 Email: felix@chelsio.com 1235 Asgeir Eiriksson 1236 Chelsio Communications, Inc. 1237 370 San Aleso Ave. 1238 Sunnyvale, CA 94085 1239 Phone: 1-408-962-3600 1240 Email: asgeir@chelsio.com 1242 Wael Noureddine 1243 Chelsio Communications, Inc. 1244 370 San Aleso Ave. 1245 Sunnyvale, CA 94085 1246 Phone: 1-408-962-3600 1247 Email: wael@chelsio.com 1249 Robert Sharp 1250 Intel Corporation 1251 1501 South Mopac, Suite 400, Mailstop: AN1-WTR1 1252 Austin, TX 78746 1253 Phone: 1-512-493-3242 1254 Email: robert.o.sharp@intel.com