idnits 2.17.1 draft-ietf-storm-rdmap-ext-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC5040]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 11, 2011) is 4645 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 1 error (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Storage Maintenance (storm) Working Group Hemal Shah 2 Internet Draft Broadcom Corporation 3 Intended status: Standards Track Felix Marti 4 Expires: January 2012 Wael Noureddine 5 Asgeir Eiriksson 6 Chelsio Communications, Inc. 7 Robert Sharp 8 Intel Corporation 9 July 11, 2011 11 RDMA Protocol Extensions 12 draft-ietf-storm-rdmap-ext-01.txt 14 Status of this Memo 16 This Internet-Draft is submitted to IETF in full conformance with 17 the provisions of BCP 78 and BCP 79. 19 Internet-Drafts are working documents of the Internet Engineering 20 Task Force (IETF). Note that other groups may also distribute 21 working documents as Internet-Drafts. The list of current Internet- 22 Drafts is at http://datatracker.ietf.org/drafts/current. 24 Internet-Drafts are draft documents valid for a maximum of six 25 months and may be updated, replaced, or obsoleted by other documents 26 at any time. It is inappropriate to use Internet-Drafts as 27 reference material or to cite them other than as "work in progress." 29 This Internet-Draft will expire on January 11, 2012. 31 Copyright Notice 33 Copyright (c) 2011 IETF Trust and the persons identified as the 34 document authors. All rights reserved. 36 This document is subject to BCP 78 and the IETF Trust's Legal 37 Provisions Relating to IETF Documents 38 (http://trustee.ietf.org/license-info) in effect on the date of 39 publication of this document. Please review these documents 40 carefully, as they describe your rights and restrictions with 41 respect to this document. Code Components extracted from this 42 document must include Simplified BSD License text as described in 43 Section 4.e of the Trust Legal Provisions and are provided without 44 warranty as described in the Simplified BSD License. 46 Abstract 48 This document specifies extensions to the IETF Remote Direct Memory 49 Access Protocol (RDMAP [RFC5040]). RDMAP provides read and write 50 services directly to applications and enables data to be transferred 51 directly into Upper Layer Protocol (ULP) Buffers without 52 intermediate data copies. The extensions specified in this document 53 provide the following capabilities and/or improvements: Atomic 54 Operations and Immediate Data. 56 Table of Contents 58 1. Introduction ................................................. 3 59 2. Requirements Language ........................................ 3 60 3. Glossary ..................................................... 3 61 4. Header Format Extensions ..................................... 5 62 4.1. RDMAP Control and Invalidate STag Fields ................ 5 63 4.2. RDMA Message Definitions ................................ 6 64 5. Atomic Operations ............................................ 7 65 5.1. Atomic Operation Details ................................ 8 66 5.1.1. FetchAdd ........................................... 8 67 5.1.2. Swap ............................................... 9 68 5.1.3. CmpSwap ........................................... 10 69 5.2. Atomic Operations ...................................... 11 70 5.2.1. Atomic Operation Request Message .................. 12 71 5.2.2. Atomic Operation Response Message ................. 15 72 5.3. Atomicity Guarantees ................................... 17 73 5.4. Atomic Operations Ordering and Completion Rules ........ 17 74 6. Immediate Data .............................................. 18 75 6.1. RDMAP Interactions with ULP for Immediate Data ......... 18 76 6.2. Immediate Data Header Format ........................... 18 77 6.3. Immediate Data or Immediate Data with SE Message ....... 19 78 6.4. Ordering and Completions ............................... 20 79 7. Ordering and Completions Table .............................. 20 80 8. Error Processing ............................................ 23 81 8.1. Errors Detected at the Local Peer ...................... 23 82 8.2. Errors Detected at the Remote Peer ..................... 24 84 9. Security Considerations ..................................... 24 85 10. IANA Considerations ........................................ 24 86 11. References ................................................. 25 87 11.1. Normative References .................................. 25 88 11.2. Informative References ................................ 25 89 12. Acknowledgments ............................................ 25 90 Appendix A. DDP Segment Formats for RDMA Messages............... 26 91 A.1. DDP Segment for Atomic Operation Request................ 26 92 A.2. DDP Segment for Atomic Response ........................ 28 93 A.3. DDP Segment for Immediate Data and Immediate Data with SE28 95 1. Introduction 97 The RDMA Protocol [RFC5040] provides capabilities for zero copy and 98 kernel bypass data communications. This document specifies the 99 following extensions to the RDMA Protocol (RDMAP): 101 o Atomic operations on remote memory locations. Support for atomic 102 operation enhances the usability of RDMAP in distributed shared 103 memory environments. 105 o Immediate Data messages allow the ULP at the sender to provide a 106 small amount of data following an RDMA Message. 108 Other RDMA transport protocols define the functionality added by 109 these extensions leading to differences in RDMA applications and/or 110 Upper Layer Protocols. Removing these differences in the transport 111 protocols simplifies these applications and ULPs and that is the 112 main motivation for the extensions specified in this document. 114 2. Requirements Language 116 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 117 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 118 document are to be interpreted as described in RFC-2119 [RFC2119]. 120 3. Glossary 122 This document is an extension of [RFC5040] and key words are defined 123 in the glossary of the referenced document. 125 Atomic Operation - is an operation that results in an execution of a 126 64-bit operation at a specific address on a remote node. The 127 consumer can use Atomic Operations to read, modify and write at the 128 destination address while at the same time guarantee that no other 129 read or write operation will occur across any other RDMAP Streams on 130 an RNIC at the Data Sink. 132 Atomic Operation Request - An RDMA Message used by the Data Source 133 to perform an Atomic Operation at the Data Sink. 135 Atomic Operation Response - An RDMA Message used by the Data Sink to 136 describe the completion of an Atomic Operation at the Data Sink. 138 CmpSwap - is an Atomic Operation that is used to compare and swap a 139 value at a specific address on a remote node. 141 FetchAdd - is an Atomic Operation that is used to atomically 142 increment a value at a specific address on a remote node. 144 Immediate Data - a small fixed size portion of data sent from the 145 Data Source to a Data Sink 147 Immediate Data Message - An RDMA Message used by the Data Source to 148 send Immediate Data to the Data Sink 150 Immediate Data with Solicited Event (SE) Message - An RDMA Message 151 used by the Data Source to send Immediate Data with Solicited Event 152 to the Data Sink 154 Requester - the sender of an RDMA Atomic Operation request. 156 Responder - the receiver of an RDMA Atomic Operation request. 158 Swap - is an Atomic Operation that is used to swap a value at a 159 specific address on a remote node. 161 4. Header Format Extensions 163 The control information of RDMA Messages is included in DDP protocol 164 [RFC5041] defined header fields, with the following new formats: 165 . Four new RDMA Messages carry additional RDMAP headers. The 166 Immediate Data operation and Immediate Data with Solicited Event 167 operation include 8 bytes of data following the RDMAP header. 168 Atomic Operations include Atomic Request or Atomic Response 169 headers following the RDMAP header. 171 4.1. RDMAP Control and Invalidate STag Fields 173 The RDMA Messages defined by this specification use all 8 bits of 174 the RDMAP Control Field. The first octet reserved for ULP use in the 175 DDP Protocol MUST be used by the RDMAP to carry the RDMAP Control 176 Field. The ordering of the bits in the first octet MUST be as shown 177 in Figure 1. 179 Figure 1 depicts the format of the DDP Control and RDMAP Control 180 fields, in the style and convention of [RFC5040]: 182 0 1 2 3 183 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 184 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 185 |T|L| Resrv | DV| RV|Rsv| Opcode| 186 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 187 | Invalidate STag | 188 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 190 Figure 1 DDP Control and RDMAP Control Fields 192 Figure 2 defines the values of RDMA Opcode field that MUST be used 193 for the RDMA Messages defined in this specification. 195 Figure 2 also defines when the STag, Tagged Offset, and Queue Number 196 fields MUST be provided for the RDMA Messages defined in this 197 specification. 199 All RDMA Messages defined in this specification MUST have: 201 The RDMA Version (RV) field: 01b. 203 Opcode field: See Figure 2. 205 Invalidate STag: MUST be set to zero by the sender, ignored by the 206 receiver. 208 -------+-----------+-------+------+-------+-----------+-------------- 209 RDMA | Message | Tagged| STag | Queue | Invalidate| Message 210 Opcode | Type | Flag | and | Number| STag | Length 211 | | | TO | | | Communicated 212 | | | | | | between DDP 213 | | | | | | and RDMAP 214 -------+-----------+-------+------+-------+-----------+-------------- 215 1000b | Immediate | 0 | N/A | 0 | N/A | Yes 216 | Data | | | | | 217 -------+-----------+------------------------------------------------- 218 1001b | Immediate | 0 | N/A | 1 | N/A | Yes 219 | Data with | | | | | 220 | SE | | | | | 221 -------+-----------+------------------------------------------------- 222 1010b | Atomic | 0 | N/A | 1 | N/A | Yes 223 | Request | | | | | 224 -------+-----------+------------------------------------------------- 225 1011b | Atomic | 0 | N/A | 1 | N/A | Yes 226 | Response | | | | | 227 -------+-----------+------------------------------------------------- 229 Figure 2 Additional RDMA Usage of DDP Fields 231 Note: N/A means Not Applicable. 233 All other DDP and RDMAP control fields MUST be set as described in 234 [RFC5040]. 236 4.2. RDMA Message Definitions 238 The following figure defines which RDMA Headers MUST be used on each 239 new RDMA Message and which new RDMA Messages are allowed to carry 240 ULP payload: 242 -------+-----------+-------------------+------------------------- 243 RDMA | Message | RDMA Header Used | ULP Message allowed in 244 Message| Type | | the RDMA Message 245 OpCode | | | 246 | | | 247 -------+-----------+-------------------+------------------------- 248 1000b | Immediate | Immediate Data | No 249 | Data | Header | 250 -------+-----------+-------------------+------------------------- 251 1001b | Immediate | Immediate Data | No 252 | Data with | Header | 253 | SE | | 254 -------+-----------+-------------------+------------------------- 255 1010b | Atomic | Atomic Request | No 256 | Request | Header | 257 -------+-----------+-------------------+------------------------- 258 1011b | Atomic | Atomic Response | No 259 | Response | Header | 260 -------+-----------+-------------------+------------------------- 261 Figure 3 RDMA Message Definitions 263 5. Atomic Operations 265 The RDMA Protocol Specification in [RFC5040] does not include 266 support for Atomic Operations which are an important building block 267 for implementing distributed shared memory. 269 This document extends the RDMA Protocol specification with a set of 270 basic Atomic Operations, and specifies their resource and ordering 271 rules. 273 Atomic operations as specified in this document execute a 64-bit 274 operation at a specified destination address on a remote node. The 275 operations atomically read, modify and write back the contents of 276 the destination address and guarantee that Atomic Operations on this 277 address by other RDMAP Streams on the same RNIC do not occur between 278 the read and the write. Atomic Operations as specified in this 279 document MAY be implemented. The discovery of whether the Atomic 280 Operations are implemented or not is outside the scope of this 281 specification and it should be handled by the ULPs or applications. 282 The advertisement of Tagged Buffer information for Atomic Operations 283 is outside the scope of this specification and it must be handled by 284 the ULPs. 286 Implementation note: It is recommended that the applications do not 287 use the buffer addresses used for Atomic Operations for other RDMA 288 operations. 290 Atomic Operations use the same remote addressing mechanism as RDMA 291 Reads and Writes. The buffer address specified in the request is in 292 the address space of the Remote Peer that the Atomic Operation is 293 targeted at. 295 5.1. Atomic Operation Details 297 The following sub-sections describe the Atomic Operations in more 298 details. 300 5.1.1. FetchAdd 302 The FetchAdd Atomic Operation requests the Responder to read a 64- 303 bit Original Remote Data value at a naturally aligned buffer address 304 in the Responder's memory, to perform FetchAdd operation on multiple 305 fields of selectable length specified by 64-bit "Add Mask", and 306 write the result back to the same virtual address. The Atomic 307 addition is performed independently on each one of these fields. A 308 bit set in the Add Mask field specifies the field boundary. FetchAdd 309 Atomic Operations MUST target buffer addresses that are naturally 310 aligned. FetchAdd Atomic Operations that target buffer addresses 311 that are not naturally aligned MUST be surfaced as errors and the 312 Responder's memory MUST NOT modified in such cases. Additionally an 313 error MUST be surfaced and a terminate message MUST be generated. 314 The setting of "Add Mask" field to 0x0000000000000000 results in 315 Atomic Add of 64-bit Original Remote Data Value and 64-bit "Add 316 Data". 318 The pseudo code below describes masked FetchAdd Atomic Operation. 320 bit_location = 1 322 carry = 0 324 Remote Data Value = 0 326 for bit = 0 to 63 328 { 330 if (bit != 0 ) bit_location = bit_location << 1 331 val1 = !(!(Original Remote Data Value & bit_location)) 333 val2 = !(!(Add Data & bit_location)) 335 sum = carry + val1 + val2 337 carry = !(!(sum & 2)) 339 sum = sum & 1 341 if (sum) 343 Remote Data Value |= bit_location 345 carry = ((carry) && (!(Add Mask & bit_location))) 347 } 349 The FetchAdd operation is performed in the endian format of the 350 target memory. The "Original Remote Data" is converted from the 351 endian format of the target memory for return and returned to the 352 Requester. The fields are in big-endian format on the wire. 354 The Requester specifies: 356 o Remote STag 358 o Remote Tagged Offset 360 o Add Data 362 o Add Mask 364 The Responder returns: 366 o Original Remote Data 368 5.1.2. Swap 370 The Swap Atomic Operation requires the Responder to read a 64-bit 371 value at a naturally aligned buffer address in the Responder's 372 memory, then to write the "Swap Data" fields into the same buffer 373 address. The "Original Remote Data" is converted from the endian 374 format of the target memory for return and returned to the 375 Requester. The fields are in big-endian format on the wire. 377 The Requester specifies: 379 o Remote STag 381 o Remote Tagged Offset 383 o Swap Data 385 The Responder returns: 387 o Original Remote Data 389 After the successful completion of Swap operation, the Responder's 390 memory at the specified buffer address MUST contain the "Swap Data" 391 field in the header. Swap Atomic Operations MUST target buffer 392 addresses that are naturally aligned. Swap Atomic Operations that 393 target buffer addresses that are not naturally aligned MUST be 394 surfaced as errors and the Responder's memory MUST NOT be modified 395 in such cases. Additionally an error MUST be surfaced and a 396 terminate message MUST be generated. 398 5.1.3. CmpSwap 400 The CmpSwap Atomic Operation requires the Responder to read a 64-bit 401 value at a naturally aligned buffer address in the Responder's 402 memory, to perform an AND logical operation using the 64 bit 403 "Compare Mask" field in the Atomic Operation Request header, then to 404 compare it with the result of a logical AND operation of the 405 "Compare Mask" and the "Compare Data" fields in the header, and, if 406 the two values are equal, to swap masked bits in the same buffer 407 address with the masked Swap Data. If the two masked compare values 408 are not equal, the contents of the Responder's memory are not 409 changed. In either case, the original value read from the buffer 410 address is converted from the endian format of the target memory for 411 return and returned to the Requester. The fields are in big-endian 412 format on the wire. 414 The Requester specifies: 416 o Remote STag 418 o Remote Tagged Offset 420 o Swap Data 421 o Swap Mask 423 o Compare Data 425 o Compare Mask 427 The Responder returns: 429 o Original Remote Data Value 431 The following pseudo code describes the masked CmpSwap operation 432 result. 434 if (!((Compare Data ^ Original Remote Data value) & Compare Mask) 436 then 438 Remote Data Value = 440 (Original Remote Data Value & ~(Swap Mask)) 442 | (Swap Data & Swap Mask) 444 else 446 Remote Data Value = Original Remote Data Value 448 After the operation, the remote data buffer MUST contain the 449 "Original Remote Data Value" (if comparison did not match) or the 450 masked "Swap Data" (if the comparison did match). CmpSwap Atomic 451 Operations MUST target buffer addresses that are naturally aligned. 452 CmpSwap Atomic Operations that target buffer addresses that are not 453 naturally aligned MUST be surfaced as errors and the remote data 454 buffer MUST NOT be modified in such cases. Additionally an error 455 MUST be surfaced and a terminate message MUST be generated. 457 5.2. Atomic Operations 459 The Atomic Operation Request and Response are RDMA Messages. An 460 Atomic Operation makes use of the DDP Untagged Buffer Model. Atomic 461 Operations use the same Queue Number as RDMA Read Requests (QN=1). 462 Reusing the same Queue Number allows the Atomic Operations to reuse 463 the same infrastructure (e.g. ORD/IRD flow control) as defined for 464 RDMA Read Requests. 466 The RDMA Message OpCode for an Atomic Request Message is 1010b. The 467 RDMA Message OpCode for an Atomic Response Message is 1011b. 469 5.2.1. Atomic Operation Request Message 471 The Atomic Operation Request Message carries an Atomic Operation 472 Header that describes the buffer address in the Responder's memory. 473 The Atomic Operation Request header immediately follows the DDP 474 header. The RDMAP layer passes to the DDP layer a RDMAP Control 475 Field. The following figure depicts the Atomic Operation Request 476 Header that MUST be used for all Atomic Operation Request Messages: 478 0 1 2 3 479 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 480 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 481 | Reserved (Not Used) |AOpCode| 482 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 483 | Request Identifier | 484 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 485 | Remote STag | 486 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 487 | Remote Tagged Offset | 488 + + 489 | | 490 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 491 | Add or Swap Data | 492 + + 493 | | 494 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 495 | Add or Swap Mask | 496 + + 497 | | 498 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 499 | Compare Data | 500 + + 501 | | 502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 | Compare Mask | 504 + + 505 | | 506 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 508 Figure 4 Atomic Operation Request Header 510 Reserved (Not Used): 28 bits 512 This field MUST be set to zero on transmit, ignored on 513 receive. 515 Atomic Operation Code (AOpCode): 4 bits. 517 See Figure 5. 519 Request Identifier: 32 bits. 521 The Request Identifier specifies a number that is used to 522 identify Atomic Operation Request Message. The use of this 523 field is implementation dependent and outside the scope of 524 this specification. 526 Remote STag: 32 bits. 528 The Remote STag identifies the Remote Peer's Tagged Buffer 529 targeted by the Atomic Operation. The Remote STag is 530 associated with the RDMAP Stream through a mechanism that is 531 outside the scope of the RDMAP specification. 533 Remote Tagged Offset: 64 bits. 535 The Remote Tagged Offset specifies the starting offset, in 536 octets, from the base of the Remote Peer's Tagged Buffer 537 targeted by the Atomic Operation. The Remote Tagged Offset MAY 538 start at an arbitrary offset. 540 Add or Swap Data: 64 bits. 542 The Add or Swap Data field specifies the 64-bit "Add Data" 543 value in an Atomic FetchAdd Operation or the 64-bit "Swap 544 Data" value in an Atomic Swap or CmpSwap Operation. 546 Add or Swap Mask: 64 bits 548 This field is used in masked Atomic Operations (FetchAdd and 549 CmpSwap) to perform a bitwise logical AND operation as specified 550 in the definition of these operations. For non-masked Atomic 551 Operations (Swap), this field MUST be set to ffffffffffffffffh on 552 transmit and ignored by the receiver. 554 Compare Data: 64 bits. 556 The Compare Data field specifies the 64-bit "Compare Data" 557 value in an Atomic CmpSwap Operation. For Atomic FetchAdd and 558 Atomic Swap operation, the Compare Data field MUST be set to 559 zero on transmit and ignored by the receiver. 561 Compare Mask: 64 bits 563 This field is used in masked Atomic Operation CmpSwap to 564 perform a bitwise logical AND operation as specified in the 565 definition of these operations. For Atomic Operations FetchAdd 566 and Swap, this field MUST be set to ffffffffffffffffh on 567 transmit and ignored by the receiver. 569 ---------+-----------+----------+----------+---------+--------- 570 Atomic | Atomic | Add or | Add or | Compare | Compare 571 Operation| Operation | Swap | Swap | Data | Mask 572 OpCode | | Data | Mask | | 573 ---------+-----------+----------+----------+---------+--------- 574 0000b | FetchAdd | Add Data | Add Mask | N/A | N/A 575 ---------+-----------+----------+----------+---------+--------- 576 0001b | Swap | Swap Data| N/A | N/A | N/A 577 ---------+-----------+----------+----------+---------+--------- 578 0010b | CmpSwap | Swap Data| Swap Mask| Valid | Valid 579 ---------+-----------+----------+----------+---------+--------- 580 0011b | | 581 to | Reserved | Not Specified 582 1111b | | 583 ---------+-----------+----------------------------------------- 585 Figure 5 Atomic Operation Message Definitions 587 The Atomic Operation Request Message has the following semantics: 589 1. An Atomic Operation Request Message MUST reference an Untagged 590 Buffer. That is, the Local Peer's RDMAP layer MUST request that 591 the DDP mark the Message as Untagged. 593 2. One Atomic Operation Request Message MUST consume one Untagged 594 Buffer. 596 3. The Responder's RDMAP layer MUST process an Atomic Operation 597 Request Message. A valid Atomic Operation Request Message MUST 598 NOT be delivered to the Responder's ULP (i.e., it is processed by 599 the RDMAP layer). 601 4. At the Responder, when an invalid Atomic Operation Request 602 Message is delivered to the Remote Peer's RDMAP layer, an error 603 is surfaced. 605 5. An Atomic Operation Request Message MUST reference the RDMA Read 606 Request Queue. That is, the Requester's RDMAP layer MUST request 607 that the DDP layer set the Queue Number field to one. 609 6. The Requester MUST pass to the DDP layer Atomic Operation Request 610 Messages in the order they were submitted by the ULP. 612 7. The Responder MUST process the Atomic Operation Request Messages 613 in the order they were sent. 615 8. If the Responder receives a valid Atomic Operation Request 616 Message, it MUST respond with a valid Atomic Operation Response 617 Message. 619 5.2.2. Atomic Operation Response Message 621 The Atomic Operation Response Message carries an Atomic Operation 622 Response Header that contains the "Original Request Identifier" and 623 "Original Remote Data Value". The Atomic Operation Response Header 624 immediately follows the DDP header. The RDMAP layer passes to the 625 DDP layer a RDMAP Control Field. The following figure depicts the 626 Atomic Operation Response header that MUST be used for all Atomic 627 Operation Response Messages: 629 0 1 2 3 630 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 631 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 632 | Original Request Identifier | 633 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 634 | Original Remote Data Value | 635 + + 636 | | 637 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 639 Figure 6 Atomic Operation Response Header 641 Original Request Identifier: 32 bits. 643 The Original Request Identifier MUST be set to the value 644 specified in the Request Identifier field that was originally 645 provided in the corresponding Atomic Operation Request 646 Message. 648 Original Remote Data Value: 64 bits. 650 The Original Remote Value specifies the original 64-bit value 651 stored at the buffer address targeted by the Atomic Operation. 653 The Atomic Operation Response Message has the following semantics: 655 1. The Atomic Operation Response Message for the associated Atomic 656 Operation Request Message travels in the opposite direction. 658 2. An Atomic Operation Response Message MUST consume an Untagged 659 Buffer. That is, the Responder RDMAP layer MUST request that the 660 DDP mark the Message as Untagged. 662 3. An Atomic Operation Response Message MUST reference the Queue 663 Number 3. That is, the Responder's RDMAP layer MUST request that 664 the DDP layer set the Queue Number field to 3. 666 4. The Responder MUST ensure that a sufficient number of Untagged 667 Buffers are available on the RDMA Read Request Queue (Queue with 668 DDP Queue Number 1) to support the maximum number of Atomic 669 Operation Requests negotiated by the ULP. 671 5. The RDMAP layer MUST Deliver the Atomic Operation Response 672 Message to the ULP. 674 6. At the Requester, when an invalid Atomic Operation Response 675 Message is delivered to the Remote Peer's RDMAP layer, an error 676 is surfaced. 678 7. The Responder RDMAP layer MUST pass Atomic Operation Response 679 Messages to the DDP layer, in the order that the Atomic Operation 680 Request Messages were received by the RDMAP layer, at the 681 Responder. 683 5.3. Atomicity Guarantees 685 Atomicity of the Read-Modify-Write (RMW) on the Responder's node by 686 the Atomic Operation MUST be assured in the presence of concurrent 687 atomic accesses by other RDMAP Streams on the same RNIC. 689 5.4. Atomic Operations Ordering and Completion Rules 691 In addition to the ordering and completion rules described in 692 [RFC5040], the following rules apply to implementations of the 693 Atomic operations. 695 1. For an Atomic operation, the contents of the Tagged Buffer at the 696 Responder MAY be indeterminate until the Atomic Operation 697 Response Message has been Delivered at the Requester. 699 2. Atomic Operation Request Messages MUST NOT start processing at 700 the Responder until they have been Delivered to RDMAP by DDP. 702 3. Atomic Operation Response Messages MAY be generated at the 703 Responder after subsequent RDMA Write Messages or Send Messages 704 have been Placed or Delivered. 706 4. Atomic Operation Response Message processing at the Responder 707 MUST be started only after the Atomic Operation Request Message 708 has been Delivered by the DDP layer (thus, all previous RDMA 709 Messages have been properly submitted for ordered Placement). 711 5. Send Messages MAY be Completed at the Responder before prior 712 incoming Atomic Operation Request Messages have completed their 713 response processing. 715 6. An Atomic Operation MUST NOT be Completed at the Requester until 716 the DDP layer Delivers the associated incoming Atomic Operation 717 Response Message. 719 7. If more than one outstanding Atomic Request Messages are 720 supported by both peers, the Atomic Operation Request Messages 721 MUST be processed in the order they were delivered by the DDP 722 layer on the Responder. Atomic Operation Response Messages MUST 723 be submitted to the DDP layer on the Responder in the order the 724 Atomic Operation Request Messages were Delivered by DDP. 726 6. Immediate Data 728 The Immediate Data operation is used in conjunction with an RDMA 729 Operation to improve ULP processing efficiency by allowing 8 bytes 730 of immediate data to be delivered with the completion of the 731 previous operation after the previous operation has been delivered 732 at the Remote Peer. 734 6.1. RDMAP Interactions with ULP for Immediate Data 736 For Immediate Data operations, the following are the interactions 737 between the RDMAP Layer and the ULP: 738 . At the Data Source: 740 . The ULP passes to the RDMAP Layer the following: 742 . Eight bytes of ULP Immediate Data 744 . When the Immediate Data operation Completes, an indication 745 of the Completion results. 747 . At the Data Sink: 749 . If the Immediate Data operation is Completed successfully, 750 the RDMAP Layer passes the following information to the ULP 751 Layer: 753 . Eight bytes of Immediate Data 755 . An Event, if the Data Sink is configured to generate an 756 Event and the RDMA Message Opcode indicates Message Type 757 Immediate Data with Solicited Event. 759 . If the Immediate Data operation is Completed in error, the 760 Data Sink RDMAP Layer will pass up the corresponding error 761 information to the Data Sink ULP and send a Terminate 762 Message to the Data Source RDMAP Layer. The Data Source 763 RDMAP Layer will then pass up the Terminate Message to the 764 ULP. 766 6.2. Immediate Data Header Format 768 The Immediate Data and Immediate Data with SE Messages carry 769 immediate data as shown in Figure 7. The RDMAP layer passes to the 770 DDP layer an RDMAP Control Field and 8 bytes of Immediate Data. The 771 first 8 bytes of the data following the DDP header contains the 772 Immediate Data. See section A.3. for the DDP segment format of an 773 Immediate Data or Immediate Data with SE Message. 775 0 1 2 3 776 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 777 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 778 | Immediate Data | 779 + + 780 | | 781 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 783 Figure 7 Immediate Data or Immediate Data with SE Message Header 785 Immediate Data: 64 bits. 786 Eight bytes of data transferred from the Requester to an 787 untagged buffer at the Responder. 789 6.3. Immediate Data or Immediate Data with SE Message 791 The Immediate Data or Immediate Data with SE Message uses the DDP 792 Untagged Buffer Model to transfer Immediate data from the Data 793 Source to the Data Sink. 794 . An Immediate Data or Immediate Data with SE Message MUST 795 reference an Untagged Buffer. That is, the Local Peer's RDMAP 796 Layer MUST request that the DDP layer mark the Message as 797 Untagged. 799 . One Immediate Data or Immediate Data with SE Message MUST consume 800 one Untagged Buffer. 802 . At the Remote Peer, the Immediate Data or Immediate Data with SE 803 Message MUST be Delivered to the Remote Peer's ULP in the order 804 they were sent. 806 . For an Immediate Data or Immediate Data with SE Message, the 807 Local Peer's RDMAP Layer MUST request that the DDP layer set the 808 Queue Number field to zero. 810 . For an Immediate Data or Immediate Data with SE Message, the 811 Local Peer's RDMAP Layer MUST request that the DDP layer transmit 812 8 bytes of data. 814 . The Local Peer MUST issue Immediate Data and Immediate Data with 815 SE Messages in the order they were submitted by the ULP. 817 . The Remote Peer MUST check that Immediate Data and Immediate Data 818 with SE Messages include exactly 8 bytes of data from the DDP 819 layer. 821 6.4. Ordering and Completions 823 Ordering and completion rules for Immediate Data are the same as 824 those for a Send operation as described in section 5.5 of RFC 5040. 826 7. Ordering and Completions Table 828 The following table summarizes the ordering relationships for Atomic 829 and Immediate Data operations from the standpoint of Local Peer issuing 830 the Operations. Note that in the table that follows, Send includes 831 Send, Send with Invalidate, Send with Solicited Event, and Send with 832 Solicited Event and Invalidate. Also note that in the table below, 833 Immediate Data includes Immediate Data and Immediate Data with 834 Solicited Event. 836 ----------+------------+-------------+-------------+------------------- 837 First | Second | Placement | Placement | Ordering 838 Operation | Operation | Guarantee at| Guarantee at| Guarantee at 839 | | Remote Peer | Local Peer | Remote Peer 840 ----------+------------+-------------+-------------+------------------- 841 Immediate | Send | No Placement| Not | Completed in 842 Data | | Guarantee | Applicable | Order 843 | | between Send| | 844 | | Payload and | | 845 | | Immediate | | 846 | | Data | | 847 ----------+------------+-------------+-------------+------------------- 848 Immediate | RDMA | No Placement| Not | Not 849 Data | Write | Guarantee | Applicable | Applicable 850 | | between RDMA| | 851 | | Write | | 852 | | Payload and | | 853 | | Immediate | | 854 | | Data | | 855 ----------+------------+-------------+-------------+------------------- 856 Immediate | RDMA | No Placement| RDMA Read | RDMA Read 857 Data | Read | Guarantee | Response | Response 858 | | between | will not be | Message will 859 | | Immediate | Placed until| not be 860 | | Data and | Immediate | generated 861 | | RDMA Read | Data is | until 862 | | Request | Placed at | Immediate Data 863 | | | Remote Peer | has been 864 | | | | Completed 865 ----------+------------+-------------+-------------+------------------- 866 Immediate | Atomic | No Placement| Atomic | Atomic 867 Data | | Guarantee | Response | Response 868 | | between | will not be | Message will 869 | | Immediate | Placed until| not be 870 | | Data and | Immediate | generated 871 | | Atomic | Data is | until 872 | | Request | Placed at | Immediate Data 873 | | | Remote Peer | has been 874 | | | | Completed 875 ----------+------------+-------------+-------------+------------------- 876 Immediate | Immediate | No Placement| Not | Completed in 877 Data or | Data | Guarantee | Applicable | Order 878 Send | | | | 879 ----------+------------+-------------+-------------+------------------- 880 RDMA Write| Immediate | No Placement| Not | Immediate Data 881 | Data | Guarantee | Applicable | is Completed 882 | | | | after RDMA 883 | | | | Write is Placed 884 | | | | and Delivered 885 ----------+------------+-------------+-------------+------------------- 886 RDMA Read | Immediate | No Placement| Immediate | Not Applicable 887 | Data | Guarantee | Data MAY be | 888 | | between | Placed | 889 | | Immediate | before | 890 | | Data and | RDMA Read | 891 | | RDMA Read | Response is | 892 | | Request | generated | 893 ----------+------------+-------------+-------------+------------------- 894 Atomic | Immediate | No Placement| Immediate | Not Applicable 895 | Data | Guarantee | Data MAY be | 896 | | between | Placed | 897 | | Immediate | before | 898 | | Data and | Atomic | 899 | | Atomic | Response is | 900 | | Request | generated | 901 ----------+------------+-------------+-------------+------------------- 902 Atomic | Send | No Placement| Send Payload| Not Applicable 903 | | Guarantee | MAY be | 904 | | between Send| Placed | 905 | | Payload and | before | 906 | | Atomic | Atomic | 907 | | Request | Response is | 908 | | | generated | 909 ----------+------------+-------------+-------------+------------------- 910 Atomic | RDMA | No Placement| RDMA Write | Not 911 | Write | Guarantee | Payload MAY | Applicable 912 | | between RDMA| be Placed | 913 | | Write | before | 914 | | Payload and | Atomic | 915 | | Atomic | Response is | 916 | | Request | generated | 917 ----------+------------+-------------+-------------+------------------- 918 Atomic | RDMA | No Placement| No Placement| RDMA Read 919 | Read | Guarantee | Guarantee | Response 920 | | between | between | Message will 921 | | Atomic | Atomic | not be 922 | | Request and | Response | generated 923 | | RDMA Read | and RDMA | until Atomic 924 | | Request | Read | Response Message 925 | | | Response | has been 926 | | | | generated 927 ----------+------------+-------------+-------------+------------------- 928 Atomic | Atomic | No Placement| No Placement| Second Atomic 929 | | Guarantee | Guarantee | Response 930 | | between two | between two | Message will 931 | | Atomic | Atomic | not be 932 | | Requests | Responses | generated 933 | | | | until first 934 | | | | Atomic Response 935 | | | | has been 936 | | | | generated 937 ----------+------------+-------------+-------------+------------------- 938 Send | Atomic | No Placement| Atomic | Atomic Response 939 | | Guarantee | Response | Message will not 940 | | between Send| will not be | be generated until 941 | | Payload and | Placed at | Send has been 942 | | Atomic | the Local | Completed 943 | | Request | Peer Until | 944 | | | Send Payload| 945 | | | is Placed | 946 | | | at the | 947 | | | Remote Peer | 948 ----------+------------+-------------+-------------+------------------- 949 RDMA | Atomic | No Placement| Atomic | Not 950 Write | | Guarantee | Response | Applicable 951 | | between RDMA| will not be | 952 | | Write | Placed at | 953 | | Payload and | the Local | 954 | | Atomic | Peer Until | 955 | | Request | Send Payload| 956 | | | is Placed | 957 | | | at the | 958 | | | Remote Peer | 959 ----------+------------+-------------+-------------+------------------- 960 RDMA | Atomic | No Placement| No Placement| Atomic Response 961 Read | | Guarantee | Guarantee | Message will 962 | | between | between | not be generated 963 | | Atomic | Atomic | until RDMA 964 | | Request and | Response | Read Response 965 | | RDMA Read | and RDMA | has been 966 | | Request | Read | generated 967 | | | Response | 968 ----------+------------+-------------+-------------+------------------- 970 8. Error Processing 972 In addition to error processing described in section 7 of [RFC5040], 973 the following rules apply for the new RDMA Messages defined in this 974 specification. 976 8.1. Errors Detected at the Local Peer 978 The Local Peer MUST send a Terminate Message for each of the 979 following cases: 981 1. For errors detected while creating an Atomic Request, Atomic 982 Response, Immediate Data, or Immediate Data with SE Message, or 983 other reasons not directly associated with an incoming Message, 984 the Terminate Message and Error code are sent instead of the 985 Message. In this case, the Error Type and Error Code fields are 986 included in the Terminate Message, but the Terminated DDP Header 987 and Terminated RDMA Header fields are set to zero. 989 2. For errors detected on an incoming Atomic Request, Atomic 990 Response, Immediate Data, or Immediate Data with Solicited Event 991 (after the Message has been Delivered by DDP), the Terminate 992 Message is sent at the earliest possible opportunity, preferably 993 in the next outgoing RDMA Message. In this case, the Error Type, 994 Error Code, and Terminated DDP Header fields are included in the 995 Terminate Message, but the Terminated RDMA Header field is set to 996 zero. 998 8.2. Errors Detected at the Remote Peer 1000 On incoming Atomic Requests, Atomic Responses, Immediate Data, and 1001 Immediate Data with Solicited Event, the following MUST be 1002 validated: 1004 . The DDP layer MUST validate all DDP Segment fields. 1006 . The RDMA OpCode MUST be valid. 1008 . The RDMA Version MUST be valid. 1010 On incoming Atomic requests the following additional validation MUST 1011 be performed: 1013 . The RDMAP layer MUST validate that the Remote Peer's Tagged 1014 Buffer address references a naturally aligned host buffer 1015 address. In the case of an error, the RDMAP layer MUST generate a 1016 Terminate Message indicating RDMA Layer Remote Operation Error 1017 with Error Code Name "Catastrophic Error, Localized to RDMAP 1018 Stream" as described in Section 4.8 of [RFC5040]. 1020 9. Security Considerations 1022 This document specifies extensions to the RDMA Protocol 1023 specification in [RFC5040], and as such the Security Considerations 1024 discussed in Section 8 of [RFC5040] apply. 1026 10. IANA Considerations 1028 This document requests no direct action from IANA. 1030 11. References 1032 11.1. Normative References 1034 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1035 Requirement Levels", BCP 14, RFC 2119, March 1997. 1037 [RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol 1038 Specification", RFC 5040, October 2007. 1040 [RFC5041] Shah, H. et al., "Direct Data Placement over Reliable 1041 Transports", RFC 5041, October 2007. 1043 11.2. Informative References 1045 12. Acknowledgments 1047 The authors would like to acknowledge the following contributors who 1048 provided valuable comments and suggestions. 1050 o Steve Wise. 1052 This document was prepared using 2-Word-v2.0.template.dot. 1054 Appendix A. DDP Segment Formats for RDMA Messages 1056 This appendix is for information only and is NOT part of the 1057 standard. It simply depicts the DDP Segment format for the various 1058 RDMA Messages. 1060 A.1. DDP Segment for Atomic Operation Request 1062 The following figure depicts an Atomic Operation Request, DDP 1063 Segment: 1065 0 1 2 3 1066 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1067 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1068 | DDP Control | RDMA Control | 1069 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1070 | Reserved (Not Used) | 1071 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1072 | DDP (Atomic Operation Request) Queue Number | 1073 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 | DDP (Atomic Operation Request) Message Sequence Number | 1075 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1076 | DDP (Atomic Operation Request) Message Offset | 1077 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1078 | Reserved (Not Used) |AOpCode| 1079 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1080 | Request Identifier | 1081 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1082 | Remote STag | 1083 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 | Remote Tagged Offset | 1085 + + 1086 | | 1087 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1088 | Add or Swap Data | 1089 + + 1090 | | 1091 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1092 | Add or Swap Mask | 1093 + + 1094 | | 1095 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1096 | Compare Data | 1097 + + 1098 | | 1099 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1100 | Compare Mask | 1101 + + 1102 | | 1103 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1105 A.2. DDP Segment for Atomic Response 1107 The following figure depicts an Atomic Operation Response, DDP 1108 Segment: 1110 0 1 2 3 1111 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1112 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1113 | DDP Control | RDMA Control | 1114 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1115 | Reserved (Not Used) | 1116 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1117 | DDP (Atomic Operation Request) Queue Number | 1118 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1119 | DDP (Atomic Operation Request) Message Sequence Number | 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1121 | DDP (Atomic Operation Request) Message Offset | 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1123 | Original Request Identifier | 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1125 | Original Remote Value | 1126 + + 1127 | | 1128 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1130 A.3. DDP Segment for Immediate Data and Immediate Data with SE 1132 The following figure depicts an Immediate Data or Immediate data 1133 with SE, DDP Segment: 1135 0 1 2 3 1136 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1138 | DDP Control | RDMA Control | 1139 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1140 | Reserved (Not Used) | 1141 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1142 | DDP (Send) Queue Number | 1143 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1144 | DDP (Send) Message Sequence Number | 1145 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1146 | DDP Message Offset | 1147 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1148 | Immediate Data | 1149 + + 1150 | | 1151 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1153 Authors' Addresses 1155 Hemal Shah 1156 Broadcom Corporation 1157 5300 California Avenue 1158 Irvine, CA 92617 1159 Phone: 1-949-926-6941 1160 Email: hemal@broadcom.com 1162 Felix Marti 1163 Chelsio Communications, Inc. 1164 370 San Aleso Ave. 1165 Sunnyvale, CA 94085 1166 Phone: 1-408-962-3600 1167 Email: felix@chelsio.com 1169 Asgeir Eiriksson 1170 Chelsio Communications, Inc. 1171 370 San Aleso Ave. 1172 Sunnyvale, CA 94085 1173 Phone: 1-408-962-3600 1174 Email: asgeir@chelsio.com 1176 Wael Noureddine 1177 Chelsio Communications, Inc. 1178 370 San Aleso Ave. 1179 Sunnyvale, CA 94085 1180 Phone: 1-408-962-3600 1181 Email: wael@chelsio.com 1183 Robert Sharp 1184 Intel Corporation 1185 1501 South Mopac, Suite 400, Mailstop: AN1-WTR1 1186 Austin, TX 78746 1187 Phone: 1-512-493-3242 1188 Email: robert.o.sharp@intel.com