idnits 2.17.1 draft-ietf-rddp-rdmap-02.txt: ** The Abstract section seems to be numbered -(2258): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding -(2710): Line appears to be too long, but this could be caused by non-ascii characters in UTF-8 encoding Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1.a on line 22. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement. ** The document seems to lack an RFC 3978 Section 5.4 (updated by RFC 4748) Copyright Line. ** The document seems to lack an RFC 3978 Section 5.4 Reference to BCP 78. ** The document seems to lack an RFC 3978 Section 5.5 (updated by RFC 4748) Disclaimer -- however, there's a paragraph with a matching beginning. Boilerplate error? ** The document seems to lack an RFC 3979 Section 5, para. 1 IPR Disclosure Acknowledgement. ** The document seems to lack an RFC 3979 Section 5, para. 2 IPR Disclosure Acknowledgement. ** The document seems to lack an RFC 3979 Section 5, para. 3 IPR Disclosure Invitation. ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: This document is an Internet-Draft and is subject to all provisions of Section 3 of RFC 3667. By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 2 instances of lines with non-ascii characters in the document. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There are 30 instances of too long lines in the document, the longest one being 1 character in excess of 72. Miscellaneous warnings: ---------------------------------------------------------------------------- == The "Author's Address" (or "Authors' Addresses") section title is misspelled. == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (September 2, 2004) is 7147 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'IPSEC' is mentioned on line 2073, but not defined == Unused Reference: 'RFC2119' is defined on line 2255, but no explicit reference was found in the text == Unused Reference: 'VERBS' is defined on line 2258, but no explicit reference was found in the text == Unused Reference: 'RFC2401' is defined on line 2278, but no explicit reference was found in the text -- Possible downref: Normative reference to a draft: ref. 'VERBS' == Outdated reference: A later version (-07) exists of draft-ietf-rddp-ddp-03 == Outdated reference: A later version (-08) exists of draft-ietf-rddp-mpa-01 ** Obsolete normative reference: RFC 2960 (ref. 'SCTP') (Obsoleted by RFC 4960) ** Obsolete normative reference: RFC 793 (ref. 'TCP') (Obsoleted by RFC 9293) == Outdated reference: A later version (-10) exists of draft-ietf-rddp-security-05 -- Obsolete informational reference (is this intentional?): RFC 2401 (Obsoleted by RFC 4301) -- Obsolete informational reference (is this intentional?): RFC 2246 (ref. 'TLS') (Obsoleted by RFC 4346) Summary: 14 errors (**), 0 flaws (~~), 11 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Remote Direct Data Placement Work Group R. Recio 3 INTERNET DRAFT IBM Corporation 4 draft-ietf-rddp-rdmap-02.txt P. Culley 5 Hewlett-Packard Company 6 D. Garcia 7 Hewlett-Packard Company 8 J. Hilland 9 Hewlett-Packard Company 11 Expires: March, 2005 September 2, 2004 13 An RDMA Protocol Specification 15 1 Status of this Memo 17 This document is an Internet-Draft and is subject to all 18 provisions of Section 3 of RFC 3667. By submitting this Internet- 19 Draft, each author represents that any applicable patent or other 20 IPR claims of which he or she is aware have been or will be 21 disclosed, and any of which he or she become aware will be 22 disclosed, in accordance with RFC 3668. 24 Internet-Drafts are working documents of the Internet Engineering 25 Task Force (IETF), its areas, and its working groups. Note that 26 other groups may also distribute working documents as Internet- 27 Drafts. 29 Internet-Drafts are draft documents valid for a maximum of six 30 months and may be updated, replaced, or obsoleted by other 31 documents at any time. It is inappropriate to use Internet-Drafts 32 as reference material or to cite them other than as "work in 33 progress." 35 The list of current Internet-Drafts can be accessed at 36 http://www.ietf.org/1id-abstracts.html The list of Internet-Draft 37 Shadow Directories can be accessed at 38 http://www.ietf.org/shadow.html. 40 2 Abstract 42 This document defines a Remote Direct Memory Access Protocol 43 (RDMAP) that operates over the Direct Data Placement Protocol (DDP 44 protocol). RDMAP provides read and write services directly to 45 applications and enables data to be transferred directly into ULP 46 Buffers without intermediate data copies. It also enables a kernel 47 bypass implementation. 49 Table of Contents 51 1 Status of this Memo.........................................1 52 2 Abstract....................................................1 53 3 Introduction................................................5 54 3.1 Architectural Goals.........................................5 55 3.2 Protocol Overview...........................................6 56 3.3 RDMAP Layering..............................................9 57 3.4 Specification Changes from the Last Version................10 58 4 Glossary...................................................12 59 4.1 General....................................................12 60 4.2 LLP........................................................13 61 4.3 Direct Data Placement (DDP)................................14 62 4.4 Remote Direct Memory Access (RDMA).........................16 63 5 ULP and Transport Attributes...............................20 64 5.1 Transport Requirements & Assumptions.......................20 65 5.2 RDMAP Interactions with the ULP............................21 66 6 Header Format..............................................25 67 6.1 RDMAP Control and Invalidate STag Field....................25 68 6.2 RDMA Message Definitions...................................28 69 6.3 RDMA Write Header..........................................29 70 6.4 RDMA Read Request Header...................................30 71 6.5 RDMA Read Response Header..................................32 72 6.6 Send Header and Send with Solicited Event Header...........32 73 6.7 Send with Invalidate Header and Send with SE and Invalidate 74 Header...........................................................32 75 6.8 Terminate Header...........................................32 76 7 Data Transfer..............................................39 77 7.1 RDMA Write Message.........................................39 78 7.2 RDMA Read Operation........................................40 79 7.2.1 RDMA Read Request Message.................................40 80 7.2.2 RDMA Read Response Message................................41 81 7.3 Send Message Type..........................................42 82 7.4 Terminate Message..........................................44 83 7.5 Ordering and Completions...................................45 84 8 RDMAP Stream Management....................................49 85 8.1 Stream Initialization......................................49 86 8.2 Stream Teardown............................................50 87 8.2.1 RDMAP Abortive Termination................................50 88 9 RDMAP Error Management.....................................52 89 9.1 RDMAP Error Surfacing......................................52 90 9.2 Errors Detected at the Remote Peer on Incoming RDMA Messages53 91 10 Security Considerations....................................55 92 10.1 Protocol-specific Security Considerations.................55 93 10.2 Using IPSec with RDMAP....................................55 94 10.3 Other Security Considerations.............................55 95 11 References.................................................60 96 11.1 Normative References......................................60 97 11.2 Informative References....................................60 98 12 Appendix...................................................61 99 12.1 DDP Segment Formats for RDMA Messages.....................61 100 12.1.1 DDP Segment for RDMA Write..............................61 101 12.1.2 DDP Segment for RDMA Read Request.......................61 102 12.1.3 DDP Segment for RDMA Read Response......................63 103 12.1.4 DDP Segment for Send and Send with Solicited Event......63 104 12.1.5 DDP Segment for Send with Invalidate and Send with SE and 105 Invalidate.......................................................64 106 12.1.6 DDP Segment for Terminate...............................65 107 12.2 Ordering and Completion Table.............................65 108 13 Authors Addresses..........................................69 109 14 Acknowledgments............................................70 110 15 Full Copyright Statement...................................73 112 Table of Figures 114 Figure 1 RDMAP Layering...........................................9 115 Figure 2 Example of MPA, DDP, and RDMAP Header Alignment over TCP10 116 Figure 3 DDP Control, RDMAP Control, and Invalidate STag Fields..26 117 Figure 4 RDMA Usage of DDP Fields................................27 118 Figure 5 RDMA Message Definitions................................29 119 Figure 6 RDMA Read Request Header Format.........................30 120 Figure 7 Terminate Header Format.................................33 121 Figure 8 Terminate Control Field.................................33 122 Figure 9 Terminate Control Field Values..........................36 123 Figure 10 Error Type to RDMA Message Mapping.....................38 124 Figure 11 RDMA Write, DDP Segment format.........................61 125 Figure 12 RDMA Read Request, DDP Segment format..................62 126 Figure 13 RDMA Read Response, DDP Segment format.................63 127 Figure 14 Send and Send with Solicited Event, DDP Segment format.64 128 Figure 15 Send with Invalidate and Send with SE and Invalidate, 129 DDP Segment......................................................64 130 Figure 16 Terminate, DDP Segment format..........................65 131 Figure 17 Operation Ordering.....................................68 133 3 Introduction 135 Today, communications over TCP/IP typically require copy 136 operations, which add latency and consume significant CPU and 137 memory resources. The Remote Direct Memory Access Protocol 138 (RDMAP) enables removal of data copy operations and enables 139 reduction in latencies by allowing a local application to read or 140 write data on a remote computer's memory with minimal demands on 141 memory bus bandwidth and CPU processing overhead, while preserving 142 memory protection semantics. 144 RDMAP is layered on top of Direct Data Placement (DDP) and uses 145 the two Buffer Models available from DDP [DDP]. 147 3.1 Architectural Goals 149 RDMAP has been designed with the following high-level 150 architectural goals: 152 * Provide a data transfer operation that allows a Local Peer to 153 transfer up to 2^32 - 1 octets directly into a previously 154 advertised buffer (i.e. Tagged buffer) located at a Remote Peer 155 without requiring a copy operation. This is referred to as the 156 RDMA Write data transfer operation. 158 * Provide a data transfer operation that allows a Local Peer to 159 retrieve up to 2^32 - 1 octets directly from a previously 160 advertised buffer (i.e. Tagged buffer) located at a Remote Peer 161 without requiring a copy operation. This is referred to as the 162 RDMA Read data transfer operation. 164 * Provide a data transfer operation that allows a Local Peer to 165 send up to 2^32 - 1 octets directly into a buffer located at a 166 Remote Peer that has not been explicitly advertised. This is 167 referred to as the Send (Send with Invalidate, Send with 168 Solicited Event, and Send with Solicited Event and Invalidate) 169 data transfer operation. 171 * Enable the local ULP to use the Send Operation Type (includes 172 Send, Send with Invalidate, Send with Solicited Event, and Send 173 with Solicited Event and Invalidate) to signal to the remote 174 ULP the Completion of all previous Messages initiated by the 175 local ULP. 177 * Provide for all Operations on a single RDMAP Stream to be 178 reliably transmitted in the order that they were submitted. 180 * Provide RDMAP capabilities independently for each Stream when 181 the LLP supports multiple data Streams within an LLP 182 connection. 184 3.2 Protocol Overview 186 RDMAP provides seven data transfer operations. Except for the RDMA 187 Read operation, each operation generates exactly one RDMA Message. 188 Following is a brief overview of the RDMA Operations and RDMA 189 Messages: 191 1. Send - A Send operation uses a Send Message to transfer data 192 from the Data Source into a buffer that has not been 193 explicitly Advertised by the Data Sink. The Send Message uses 194 the DDP Untagged Buffer Model to transfer the ULP Message into 195 the Data Sink's Untagged Buffer. 197 2. Send with Invalidate - A Send with Invalidate operation uses a 198 Send with Invalidate Message to transfer data from the Data 199 Source into a buffer that has not been explicitly Advertised 200 by the Data Sink. The Send with Invalidate Message includes 201 all functionality of the Send Message, with one addition: an 202 STag field is included in the Send With Invalidate Message and 203 after the message has been Placed and Delivered at the Data 204 Sink the remote peer's buffer identified by the STag can no 205 longer be accessed remotely until the remote peer's ULP re- 206 enables access and Advertises the buffer. 208 3. Send with Solicited Event (Send with SE) - A Send with 209 Solicited Event operation uses a Send with Solicited Event 210 Message to transfer data from the Data Source into an Untagged 211 Buffer at the Data Sink. The Send with Solicited Event Message 212 is similar to the Send Message, with one addition: when the 213 Send with Solicited Event Message has been Placed and 214 Delivered, an Event may be generated at the recipient, if the 215 recipient is configured to generate such an Event. 217 4. Send with Solicited Event and Invalidate (Send with SE and 218 Invalidate) - A Send with Solicited Event and Invalidate 219 operation uses a Send with Solicited Event and Invalidate 220 Message to transfer data from the Data Source into a buffer 221 that has not been explicitly Advertised by the Data Sink. The 222 Send with Solicited Event and Invalidate Message is similar to 223 the Send with Invalidate Message, with one addition: when the 224 Send with Solicited Event and Invalidate Message has been 225 Placed and Delivered, an Event may be generated at the 226 recipient, if the recipient is configured to generate such an 227 Event. 229 5. Remote Direct Memory Access Write - An RDMA Write operation 230 uses an RDMA Write Message to transfer data from the Data 231 Source to a previously advertised buffer at the Data Sink. 233 The ULP at the Remote Peer, which in this case is the Data 234 Sink, enables the Data Sink Tagged Buffer for access and 235 Advertises the buffer's size (length), location (Tagged 236 Offset), and Steering Tag (STag) to the Data Source through a 237 ULP specific mechanism. The ULP at the Local Peer, which in 238 this case is the Data Source, initiates the RDMA Write 239 operation. The RDMA Write Message uses the DDP Tagged Buffer 240 Model to transfer the ULP Message into the Data Sink's Tagged 241 Buffer. Note: the STag associated with the Tagged Buffer 242 remains valid until the ULP at the Remote Peer invalidates it 243 or the ULP at the Local Peer invalidates it through a Send 244 with Invalidate or Send with Solicited Event and Invalidate. 246 6. Remote Direct Memory Access Read - The RDMA Read operation 247 transfers data to a Tagged Buffer at the Local Peer, which in 248 this case is the Data Sink, from a Tagged Buffer at the Remote 249 Peer, which in this case is the Data Source. The ULP at the 250 Data Source enables the Data Source Tagged Buffer for access 251 and Advertises the buffer's size (length), location (Tagged 252 Offset), and Steering Tag (STag) to the Data Sink through a 253 ULP specific mechanism. The ULP at the Data Sink enables the 254 Data Sink Tagged Buffer for access and initiates the RDMA Read 255 operation. The RDMA Read operation consists of a single RDMA 256 Read Request Message and a single RDMA Read Response Message, 257 and the latter may be segmented into multiple DDP Segments. 259 The RDMA Read Request Message uses the DDP Untagged Buffer 260 Model to Deliver the STag, starting Tagged Offset and length 261 for both the Data Source and Data Sink Tagged Buffers to the 262 remote peer's RDMA Read Request Queue. 264 The RDMA Read Response Message uses the DDP Tagged Buffer 265 Model to Deliver the Data Source's Tagged Buffer to the Data 266 Sink, without any involvement from the ULP at the Data Source. 268 Note: the Data Source STag associated with the Tagged Buffer 269 remains valid until the ULP at the Data Source invalidates it 270 or the ULP at the Data Sink invalidates it through a Send with 271 Invalidate or Send with Solicited Event and Invalidate. The 272 Data Sink STag associated with the Tagged Buffer remains valid 273 until the ULP at the Data Sink invalidates it. 275 7. Terminate - A Terminate operation uses a Terminate Message to 276 transfer to the Remote Peer information associated with an 277 error that occurred at the Local Peer. The Terminate Message 278 uses the DDP Untagged Buffer Model to transfer the Message 279 into the Data Sink's Untagged Buffer. 281 3.3 RDMAP Layering 283 RDMAP is dependent on DDP, subject to the requirements defined in 284 section 5.1 Transport Requirements & Assumptions. Figure 1 RDMAP 285 Layering depicts the relationship between Upper Layer Protocols 286 (ULPs), RDMAP, DDP protocol, the framing layer, and the transport 287 For LLP protocol definitions of each LLP, see [MPA], [TCP], and 288 [SCTP]. 290 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 291 | | 292 | Upper Layer Protocol (ULP) | 293 | | 294 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 295 | | 296 | RDMAP | 297 | | 298 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 299 | | 300 | DDP protocol | 301 | | 302 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 303 | | | 304 | MPA | | 305 | | | 306 +-+-+-+-+-+-+-+-+-+ SCTP | 307 | | | 308 | TCP | | 309 | | | 310 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 311 Figure 1 RDMAP Layering 313 If RDMAP is layered over DDP/MPA/TCP, then the respective headers 314 and ULP Payload are arranged as follows (Note: For clarity, MPA 315 header and CRC fields are included but MPA markers are not shown): 317 0 1 2 3 318 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 319 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 320 | | 321 // TCP Header // 322 | | 323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 324 | MPA Header | | 325 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 326 | | 327 // DDP Header // 328 | | 329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 330 | | 331 // RDMA Header // 332 | | 333 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 334 | | 335 // ULP Payload // 336 | (shown with no pad bytes) | 337 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 338 | MPA CRC | 339 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 340 Figure 2 Example of MPA, DDP, and RDMAP Header Alignment over TCP 342 3.4 Specification Changes from the Last Version 344 The following major changes (vs typos) were made from the last 345 version: 347 * Section 6.8 - Explicitly defined the bit numbers for the three 348 header control bits. 350 * Section 8.1 - Stated the typical Stream initialization to be: 351 RDMA mode is entered some time after the LLP Stream is 352 initialized. 354 * Section 10 - Update reference to security document. 356 * Section 10 - Fixed Send with Solicited Event and Invalidate 357 reference. 359 * Section 12.1 - MPA and DDP references were changed to reflect 360 the released specifications and accurate titles. 362 * Section 12.1 - Reference for RDMA Protocol Verbs was changed to 363 reflect the released specification and accurate title. 365 4 Glossary 367 4.1 General 369 Advertisement (Advertised, Advertise, Advertisements, Advertises) 370 - the act of informing a Remote Peer that a local RDMA Buffer 371 is available to it. A Node makes available an RDMA Buffer for 372 incoming RDMA Read or RDMA Write access by informing its 373 RDMA/DDP peer of the Tagged Buffer identifiers (STag, base 374 address, and buffer length). This advertisement of Tagged 375 Buffer information is not defined by RDMA/DDP and is left to 376 the ULP. A typical method would be for the Local Peer to embed 377 the Tagged Buffer's Steering Tag, base address, and length in 378 a Send Message destined for the Remote Peer. 380 Data Sink - The peer receiving a data payload. Note that the Data 381 Sink can be required to both send and receive RDMA/DDP 382 Messages to transfer a data payload. 384 Data Source - The peer sending a data payload. Note that the Data 385 Source can be required to both send and receive RDMA/DDP 386 Messages to transfer a data payload. 388 Data Delivery (Delivery, Delivered, Delivers) - Delivery is 389 defined as the process of informing the ULP or consumer that a 390 particular Message is available for use. This is specifically 391 different from "Placement", which may generally occur in any 392 order, while the order of "Delivery" is strictly defined. See 393 "Data Placement". 395 Fabric - The collection of links, switches, and routers that 396 connect a set of Nodes with RDMA/DDP protocol implementations. 398 Fence (Fenced, Fences) - To block the current RDMA Operation from 399 executing until prior RDMA Operations have Completed. 401 iWARP - A suite of wire protocols comprised of RDMAP, DDP, and 402 MPA. The iWARP protocol suite may be layered above TCP, SCTP, 403 or other transport protocols. 405 Local Peer - The RDMA/DDP protocol implementation on the local end 406 of the connection. Used to refer to the local entity when 407 describing a protocol exchange or other interaction between 408 two Nodes. 410 Node - A computing device attached to one or more links of a 411 Fabric (network). A Node in this context does not refer to a 412 specific application or protocol instantiation running on the 413 computer. A Node may consist of one or more RNICs installed in 414 a host computer. 416 Remote Peer - The RDMA/DDP protocol implementation on the opposite 417 end of the connection. Used to refer to the remote entity when 418 describing protocol exchanges or other interactions between 419 two Nodes. 421 RNIC - RDMA Network Interface Controller. In this context, this 422 would be a network I/O adapter or embedded controller with 423 iWARP and verbs functionality. 425 RNIC Interface (RI) - The presentation of the RNIC to the verbs 426 Consumer as implemented through the combination of the RNIC 427 and the RNIC driver. 429 ULP - Upper Layer Protocol. The protocol layer above the protocol 430 layer currently being referenced. The ULP for RDMA/DDP is 431 expected to be an OS, Application, adaptation layer, or 432 proprietary device. The RDMA/DDP documents do not specify a 433 ULP - they provide a set of semantics that allow a ULP to be 434 designed to utilize RDMA/DDP. 436 ULP Payload - The ULP data that is contained within a single 437 protocol segment or packet (e.g. a DDP Segment). 439 Verbs - An abstract description of the functionality of a RNIC 440 Interface. The OS may expose some or all of this functionality 441 via one or more APIs to applications. The OS will also use 442 some of the functionality to manage the RNIC Interface. 444 4.2 LLP 446 LLP - Lower Layer Protocol. The protocol layer beneath the 447 protocol layer currently being referenced. For example, for 448 DDP the LLP is SCTP, MPA, or other transport protocols. For 449 RDMA, the LLP is DDP. 451 LLP Connection - Corresponds to an LLP transport-level connection 452 between the peer LLP layers on two nodes. 454 LLP Stream - Corresponds to a single LLP transport-level Stream 455 between the peer LLP layers on two Nodes. One or more LLP 456 Streams may map to a single transport-level LLP connection. 457 For transport protocols that support multiple Streams per 458 connection (e.g. SCTP), a LLP Stream corresponds to one 459 transport-level Stream. 461 MULPDU - Maximum ULPDU. The current maximum size of the record 462 that is acceptable for DDP to pass to the LLP for 463 transmission. 465 ULPDU - Upper Layer Protocol Data Unit. The data record defined 466 by the layer above MPA. 468 4.3 Direct Data Placement (DDP) 470 Data Placement (Placement, Placed, Places) - For DDP, this term is 471 specifically used to indicate the process of writing to a data 472 buffer by a DDP implementation. DDP Segments carry Placement 473 information, which may be used by the receiving DDP 474 implementation to perform Data Placement of the DDP Segment 475 ULP Payload. See "Data Delivery". 477 DDP Abortive Teardown - The act of closing a DDP Stream without 478 attempting to Complete in-progress and pending DDP Messages. 480 DDP Graceful Teardown - The act of closing a DDP Stream such that 481 all in-progress and pending DDP Messages are allowed to 482 Complete successfully. 484 DDP Control Field - a fixed 16-bit field in the DDP Header. The 485 DDP Control Field contains an 8-bit field whose contents are 486 reserved for use by the ULP. 488 DDP Header - The header present in all DDP segments. The DDP 489 Header contains control and Placement fields that are used to 490 define the final Placement location for the ULP payload 491 carried in a DDP Segment. 493 DDP Message - A ULP defined unit of data interchange, which is 494 subdivided into one or more DDP segments. This segmentation 495 may occur for a variety of reasons, including segmentation to 496 respect the maximum segment size of the underlying transport 497 protocol. 499 DDP Segment - The smallest unit of data transfer for the DDP 500 protocol. It includes a DDP Header and ULP Payload (if 501 present). A DDP Segment should be sized to fit within the 502 underlying transport protocol MULPDU. 504 DDP Stream - a sequence of DDP Messages whose ordering is defined 505 by the LLP. For SCTP, a DDP Stream maps directly to an SCTP 506 Stream. For MPA, a DDP Stream maps directly to a TCP 507 connection and a single DDP Stream is supported. Note that 508 DDP has no ordering guarantees between DDP Streams. 510 Direct Data Placement - A mechanism whereby ULP data contained 511 within DDP Segments may be Placed directly into its final 512 destination in memory without processing of the ULP. This may 513 occur even when the DDP Segments arrive out of order. Out of 514 order Placement support may require the Data Sink to implement 515 the LLP and DDP as one functional block. 517 Direct Data Placement Protocol (DDP) - Also, a wire protocol that 518 supports Direct Data Placement by associating explicit memory 519 buffer placement information with the LLP payload units. 521 Message Offset (MO) - For the DDP Untagged Buffer Model, specifies 522 the offset, in bytes, from the start of a DDP Message. 524 Message Sequence Number (MSN) - For the DDP Untagged Buffer Model, 525 specifies a sequence number that is increasing with each DDP 526 Message. 528 Queue Number (QN) - For the DDP Untagged Buffer Model, identifies 529 a destination Data Sink queue for a DDP Segment. 531 Steering Tag - An identifier of a Tagged Buffer on a Node, valid 532 as defined within a protocol specification. 534 STag - Steering Tag 535 Tagged Buffer - A buffer that is explicitly Advertised to the 536 Remote Peer through exchange of an STag, Tagged Offset, and 537 length. 539 Tagged Buffer Model - A DDP data transfer model used to transfer 540 Tagged Buffers from the Local Peer to the Remote Peer. 542 Tagged DDP Message - A DDP Message that targets a Tagged Buffer. 544 Tagged Offset (TO) - The offset within a Tagged Buffer on a Node. 546 Untagged Buffer - A buffer that is not explicitly Advertised to 547 the Remote Peer. 549 Untagged Buffer Model - A DDP data transfer model used to transfer 550 Untagged Buffers from the Local Peer to the Remote Peer. 552 Untagged DDP Message - A DDP Message that targets an Untagged 553 Buffer. 555 4.4 Remote Direct Memory Access (RDMA) 557 Event - An indication provided by the RDMAP Layer to the ULP to 558 indicate a Completion or other condition requiring immediate 559 attention. 561 Invalidate STag - A mechanism used to prevent the Remote Peer from 562 reusing a previous explicitly Advertised STag, until the Local 563 Peer makes it available through a subsequent explicit 564 Advertisement. The STag cannot be accessed remotely until it 565 is explicit Advertised again. 567 RDMA Completion (Completion, Completed, Complete, Completes) - For 568 RDMA, Completion is defined as the process of informing the 569 ULP that a particular RDMA Operation has performed all 570 functions specified for the RDMA Operations, including 571 Placement and Delivery. The Completion semantic of each RDMA 572 Operation is distinctly defined. 574 RDMA Message - A data transfer mechanism used to fulfill an RDMA 575 Operation. 577 RDMA Operation - A sequence of RDMA Messages, including control 578 Messages, to transfer data from a Data Source to a Data Sink. 579 The following RDMA Operations are defined - RDMA Writes, RDMA 580 Read, Send, Send with Invalidate, Send with Solicited Event, 581 Send with Solicited Event and Invalidate, and Terminate. 583 RDMA Protocol (RDMAP) - A wire protocol that supports RDMA 584 Operations to transfer ULP data between a Local Peer and the 585 Remote Peer. 587 RDMAP Abortive Termination (Termination, Terminated, Terminate, 588 Terminates) - The act of closing an RDMAP Stream without 589 attempting to Complete in-progress and pending RDMA 590 Operations. 592 RDMAP Graceful Termination - The act of closing an RDMAP Stream 593 such that all in-progress and pending RDMA Operations are 594 allowed to Complete successfully. 596 RDMA Read - An RDMA Operation used by the Data Sink to transfer 597 the contents of a source RDMA buffer from the Remote Peer to 598 the Local Peer. An RDMA Read operation consists of a single 599 RDMA Read Request Message and a single RDMA Read Response 600 Message. 602 RDMA Read Request - An RDMA Message used by the Data Sink to 603 request the Data Source to transfer the contents of an RDMA 604 buffer. The RDMA Read Request Message describes both the Data 605 Source and Data Sink RDMA buffers. 607 RDMA Read Request Queue - The queue used for processing RDMA Read 608 Requests. The RDMA Read Request Queue has a DDP Queue Number 609 of 1. 611 RDMA Read Response - An RDMA Message used by the Data Source to 612 transfer the contents of an RDMA buffer to the Data Sink, in 613 response to an RDMA Read Request. The RDMA Read Response 614 Message only describes the data sink RDMA buffer. 616 RDMAP Stream - An association between a pair of RDMAP 617 implementations, possibly on different Nodes, which transfer 618 ULP data using RDMA Operations. There may be multiple RDMAP 619 Streams on a single Node. An RDMAP Stream maps directly to a 620 single DDP Stream. 622 RDMA Write - An RDMA Operation that transfers the contents of a 623 source RDMA Buffer from the Local Peer to a destination RDMA 624 Buffer at the Remote Peer using RDMA. The RDMA Write Message 625 only describes the Data Sink RDMA buffer. 627 Remote Direct Memory Access (RDMA) - A method of accessing memory 628 on a remote system in which the local system specifies the 629 remote location of the data to be transferred. Employing a 630 RNIC in the remote system allows the access to take place 631 without interrupting the processing of the CPU(s) on the 632 system. 634 Send - An RDMA Operation that transfers the contents of a ULP 635 Buffer from the Local Peer to an Untagged Buffer at the Remote 636 Peer. 638 Send Message Type - A Send Message, Send with Invalidate Message, 639 Send with Solicited Event Message, or Send with Solicited 640 Event and Invalidate Message. 642 Send Operation Type - A Send Operation, Send with Invalidate 643 Operation, Send with Solicited Event Operation, or Send with 644 Solicited Event and Invalidate Operation. 646 Solicited Event (SE) - A facility by which an RDMA Operation 647 sender may cause an Event to be generated at the recipient, if 648 the recipient is configured to generate such an Event, when a 649 Send with Solicited Event or Send with Solicited Event and 650 Invalidate Message is received. Note: The Local Peer's ULP 651 can use the Solicited Event mechanism to ensure that Messages 652 designated as important to the ULP are handled in an 653 expeditious manner by the Remote Peer's ULP. The ULP at the 654 Local Peer can indicate a given Send Message Type is important 655 by using the Send with Solicited Event Message or Send with 656 Solicited Event and Invalidate Message. The ULP at the Remote 657 Peer can choose to only be notified when valid Send with 658 Solicited Event Messages and/or Send with Solicited Event and 659 Invalidate Messages arrive and handle other valid incoming 660 Send Messages or Send with Invalidate Messages at its leisure. 662 Terminate - An RDMA Message used by a Node to pass an error 663 indication to the peer Node on an RDMAP Stream. This operation 664 is for RDMAP use only. 666 ULP Buffer - A buffer owned above the RDMAP Layer and advertised 667 to the RDMAP Layer either as a Tagged Buffer or an Untagged 668 ULP Buffer. 670 ULP Message - The ULP data that is handed to a specific protocol 671 layer for transmission. Data boundaries are preserved as they 672 are transmitted through iWARP. 674 5 ULP and Transport Attributes 676 5.1 Transport Requirements & Assumptions 678 RDMAP MUST be layered on top of the Direct Data Placement Protocol 679 [DDP]. 681 RDMAP requires the following DDP support: 683 * RDMAP uses three queues for Untagged Buffers: 685 * Queue Number 0 (used by RDMAP for Send, Send with 686 Invalidate, Send with Solicited Event, and Send with 687 Solicited Event and Invalidate operations). 689 * Queue Number 1 (used by RDMAP for RDMA Read operations). 691 * Queue Number 2 (used by RDMAP for Terminate operations). 693 * DDP maps a single RDMA Message to a single DDP Message. 695 * DDP uses the STag and Tagged Offset provided by the RDMAP for 696 Tagged Buffer Messages (i.e. RDMA Write and RDMA Read 697 Response). 699 * When the DDP layer Delivers an Untagged DDP Message to the 700 RDMAP layer, DDP provides the length of the DDP Message. This 701 ensures that RDMAP does not have to carry a length field in its 702 header. 704 * When the RDMAP layer provides an RDMA Message to the DDP Layer, 705 DDP must insert the RsvdULP field value provided by the RDMAP 706 Layer into the associated DDP Message. 708 * When the DDP layer Delivers a DDP Message to the RDMAP layer, 709 DDP provides the RsvdULP field. 711 * The RsvdULP field must be 1 octet for DDP Tagged Messages and 5 712 octets for DDP Untagged Messages. 714 * DDP propagates to RDMAP all operation or protection errors 715 (used by RDMAP Terminate) and, when appropriate, the DDP Header 716 fields of the DDP Segment that encountered the error. 718 * If an RDMA Operation is aborted by DDP or a lower layer, the 719 contents of the Data Sink buffers associated with the operation 720 are considered indeterminate. 722 * DDP in conjunction with the lower layers provide reliable, in- 723 order Delivery. 725 5.2 RDMAP Interactions with the ULP 727 RDMAP provides the ULP with access to the following RDMA 728 Operations as defined in this specification: 730 * Send 732 * Send with Solicited Event 734 * Send with Invalidate 736 * Send with Solicited Event and Invalidate 738 * RDMA Write 740 * RDMA Read 742 For Send Operation Types, the following are the interactions 743 between the RDMAP Layer and the ULP: 745 * At the Data Source: 747 * The ULP passes to the RDMAP Layer the following: 749 * ULP Message Length 751 * ULP Message 753 * An indication of the Send Operation Type, where the 754 valid types are: Send, Send with Solicited Event, Send 755 with Invalidate, or Send with Solicited Event and 756 Invalidate. 758 * An Invalidate STag, if the Send Operation Type was 759 Send with Invalidate or Send with Solicited Event and 760 Invalidate. 762 * When the Send Operation Type Completes, an indication of 763 the Completion results. 765 * At the Data Sink: 767 * If the Send Operation Type Completed successfully, the 768 RDMAP Layer passes the following information to the ULP 769 Layer: 771 * ULP Message Length 773 * ULP Message 775 * An Event, if the Data Sink is configured to generate 776 an Event. 778 * An Invalidated STag, if the Send Operation Type was 779 Send with Invalidate or Send with Solicited Event and 780 Invalidate. 782 * If the Send Operation Type Completed in error, the Data 783 Sink RDMAP Layer will pass up the corresponding error 784 information to the Data Sink ULP and send a Terminate 785 Message to the Data Source RDMAP Layer. The Data Source 786 RDMAP Layer will then pass up the Terminate Message to the 787 ULP. 789 For RDMA Write Operations, the following are the interactions 790 between the RDMAP Layer and the ULP: 792 * At the Data Source: 794 * The ULP passes to the RDMAP Layer the following: 796 * ULP Message Length 798 * ULP Message 800 * Data Sink STag 802 * Data Sink Tagged Offset 804 * When the RDMA Write Operation Completes, an indication of 805 the Completion results. 807 * At the Data Sink: 809 * If the RDMA Write completed successfully, the RDMAP Layer 810 does not Deliver the RDMA Write to the ULP. It does Place 811 the ULP Message transferred through the RDMA Write Message 812 into the ULP Buffer. 814 * If the RDMA Write completed in error, the Data Sink RDMAP 815 Layer will pass up the corresponding error information to 816 the Data Sink ULP and send a Terminate Message to the Data 817 Source RDMAP Layer. The Data Source RDMAP Layer will then 818 pass up the Terminate Message to the ULP. 820 For RDMA Read Operations, the following are the interactions 821 between the RDMAP Layer and the ULP: 823 * At the Data Sink: 825 * The ULP passes to the RDMAP Layer the following: 827 * ULP Message Length 829 * Data Source STag 831 * Data Sink STag 833 * Data Source Tagged Offset 835 * Data Sink Tagged Offset 837 * When the RDMA Read Operation Completes, an indication of 838 the Completion results. 840 * At the Data Source: 842 * If no error occurred while processing the RDMA Read 843 Request, the Data Source will not pass up any information 844 to the ULP. 846 * If an error occurred while processing the RDMA Read 847 Request, the Data Source RDMAP Layer will pass up the 848 corresponding error information to the Data Source ULP and 849 send a Terminate Message to the Data Sink RDMAP Layer. The 850 Data Sink RDMAP Layer will then pass up the Terminate 851 Message to the ULP. 853 For STags made available to the RDMAP Layer, following are the 854 interactions between the RDMAP Layer and the ULP: 856 * If the ULP enables an STag, the ULP passes to the RDMAP Layer 857 the: 859 * STag; 861 * range of Tagged Offsets that are associated with a given 862 STag; 864 * remote access rights (read, write, or read and write) 865 associated with a given, valid STag; and 867 * association between a given STag and a given RDMAP Stream. 869 * If the ULP disables an STag, the ULP passes to the RDMAP Layer 870 the STag. 872 If an error occurs at the RDMAP Layer, the RDMAP Layer may pass 873 back error information (e.g. the content of a Terminate Message) 874 to the ULP. 876 6 Header Format 878 The control information of RDMA Messages is included in DDP 879 protocol defined header fields, with the following exceptions: 881 * The first octet reserved for ULP usage on all DDP Messages in 882 the DDP Protocol (i.e. the RsvdULP Field) is used by RDMAP to 883 carry the RDMA Message Opcode and the RDMAP version. This octet 884 is known as the RDMAP Control Field in this specification. For 885 Send with Invalidate and Send with Solicited Event and 886 Invalidate, RDMAP uses the second through fifth octets provided 887 by DDP on Untagged DDP Messages to carry the STag that will be 888 Invalidated. 890 * The RDMA Message length is passed by the RDMAP layer to the DDP 891 layer on all outbound transfers. 893 * For RDMA Read Request Messages, the RDMA Read Message Size is 894 included in the RDMA Read Request Header. 896 * The RDMA Message length is passed to the RDMAP Layer by the DDP 897 layer on inbound Untagged Buffer transfers. 899 * Two RDMA Messages carry additional RDMAP headers. The RDMA Read 900 Request carries the Data Sink and Data Source buffer 901 descriptions, including buffer length. The Terminate carries 902 additional information associated with the error that caused 903 the Terminate. 905 6.1 RDMAP Control and Invalidate STag Field 907 The version of RDMAP defined by this specification uses all 8 bits 908 of the RDMAP Control Field. The first octet reserved for ULP use 909 in the DDP Protocol MUST be used by the RDMAP to carry the RDMAP 910 Control Field. The ordering of the bits in the first octet MUST be 911 as defined in Figure 3 DDP Control, RDMAP Control, and Invalidate 912 STag Field. For Send with Invalidate and Send with Solicited Event 913 and Invalidate, the second through fifth octets of the DDP RsvdULP 914 field MUST be used by RDMAP to carry the Invalidate STag. Figure 3 915 DDP Control, RDMAP Control, and Invalidate STag Field depicts the 916 format of the DDP Control and RDMAP Control fields. (Note: In 917 Figure 3 DDP Control, RDMAP Control, and Invalidate STag Field, 918 the DDP Header is offset by 16 bits to accommodate the MPA header 919 defined in [MPA]. The MPA header is only present if DDP is layered 920 on top of MPA.) 922 0 1 2 3 923 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 924 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 925 |T|L| Resrv | DV| RV|Rsv| Opcode| 926 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 927 | Invalidate STag | 928 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 929 Figure 3 DDP Control, RDMAP Control, and Invalidate STag Fields 931 All RDMA Messages handed by the RDMAP Layer to the DDP layer MUST 932 define the value of the Tagged flag in the DDP Header. Figure 4 933 RDMA Usage of DDP Fields MUST be used to define the value of the 934 Tagged flag that is handed to the DDP Layer for each RDMA Message. 936 Figure 4 RDMA Usage of DDP Fields defines the value of the RDMA 937 Opcode field that MUST be used for each RDMA Message. 939 Figure 4 RDMA Usage of DDP Fields defines when the STag, Queue 940 Number, and Tagged Offset fields MUST be provided for each RDMA 941 Message. 943 For this version of the RDMAP, all RDMA Messages MUST have: 945 * Bits 24-25; RDMA Version field: 01b. 947 * Bits 26-27; Reserved. MUST be set to zero by sender, ignored by 948 the receiver. 950 * Bits 28-31; OpCode field: see Figure 4 RDMA Usage of DDP 951 Fields. 953 * Bits 32-63; Invalidate STag. However, this field is only valid 954 for Send with Invalidate and Send with Solicited Event and 955 Invalidate Messages (see Figure 4 RDMA Usage of DDP Fields). 956 For Send, Send with Solicited Event, RDMA Read Request, and 957 Terminate, the Invalidate STag field MUST be set to zero on 958 transmit and ignored by the receiver. 960 -------+-----------+-------+------+-------+-----------+-------------- 961 RDMA | Message | Tagged| STag | Queue | Invalidate| Message 962 Message| Type | Flag | and | Number| STag | Length 963 OpCode | | | TO | | | Communicated 964 | | | | | | between DDP 965 | | | | | | and RDMAP 966 -------+-----------+-------+------+-------+-----------+-------------- 967 0000b | RDMA Write| 1 | Valid| N/A | N/A | Yes 968 | | | | | | 969 -------+-----------+-------+------+-------+-----------+-------------- 970 0001b | RDMA Read | 0 | N/A | 1 | N/A | Yes 971 | Request | | | | | 972 -------+-----------+-------+------+-------+-----------+-------------- 973 0010b | RDMA Read | 1 | Valid| N/A | N/A | Yes 974 | Response | | | | | 975 -------+-----------+-------+------+-------+-----------+-------------- 976 0011b | Send | 0 | N/A | 0 | N/A | Yes 977 | | | | | | 978 -------+-----------+-------+------+-------+-----------+-------------- 979 0100b | Send with | 0 | N/A | 0 | Valid | Yes 980 | Invalidate| | | | | 981 -------+-----------+-------+------+-------+-----------+-------------- 982 0101b | Send with | 0 | N/A | 0 | N/A | Yes 983 | SE | | | | | 984 -------+-----------+-------+------+-------+-----------+-------------- 985 0110b | Send with | 0 | N/A | 0 | Valid | Yes 986 | SE and | | | | | 987 | Invalidate| | | | | 988 -------+-----------+-------+------+-------+-----------+-------------- 989 0111b | Terminate | 0 | N/A | 2 | N/A | Yes 990 | | | | | | 991 -------+-----------+-------+------+-------+-----------+-------------- 992 1000b | | 993 to | Reserved | Not Specified 994 1111b | | 995 -------+-----------+------------------------------------------------- 996 Figure 4 RDMA Usage of DDP Fields 998 Note: N/A means Not Applicable. 1000 6.2 RDMA Message Definitions 1002 The following figure defines which RDMA Headers MUST be used on 1003 each RDMA Message and which RDMA Messages are allowed to carry ULP 1004 payload: 1006 -------+-----------+-------------------+------------------------- 1007 RDMA | Message | RDMA Header Used | ULP Message allowed in 1008 Message| Type | | the RDMA Message 1009 OpCode | | | 1010 | | | 1011 -------+-----------+-------------------+------------------------- 1012 0000b | RDMA Write| None | Yes 1013 | | | 1014 -------+-----------+-------------------+------------------------- 1015 0001b | RDMA Read | RDMA Read Request | No 1016 | Request | Header | 1017 -------+-----------+-------------------+------------------------- 1018 0010b | RDMA Read | None | Yes 1019 | Response | | 1020 -------+-----------+-------------------+------------------------- 1021 0011b | Send | None | Yes 1022 | | | 1023 -------+-----------+-------------------+------------------------- 1024 0100b | Send with | None | Yes 1025 | Invalidate| | 1026 -------+-----------+-------------------+------------------------- 1027 0101b | Send with | None | Yes 1028 | SE | | 1029 -------+-----------+-------------------+------------------------- 1030 0110b | Send with | None | Yes 1031 | SE and | | 1032 | Invalidate| | 1033 -------+-----------+-------------------+------------------------- 1034 0111b | Terminate | Terminate Header | No 1035 | | | 1036 -------+-----------+-------------------+------------------------- 1037 1000b | | 1038 to | Reserved | Not Specified 1039 1111b | | 1040 -------+-----------+-------------------+------------------------- 1041 Figure 5 RDMA Message Definitions 1043 6.3 RDMA Write Header 1045 The RDMA Write Message does not include an RDMAP header. The RDMAP 1046 layer passes to the DDP layer an RDMAP Control Field. The RDMA 1047 Write Message is fully described by the DDP Headers of the DDP 1048 Segments associated with the Message. 1050 See section 12 Appendix for a description of the DDP Segment 1051 format associated with RDMA Write Messages. 1053 6.4 RDMA Read Request Header 1055 The RDMA Read Request Message carries an RDMA Read Request Header 1056 that describes the Data Sink and Data Source Buffers used by the 1057 RDMA Read operation. The RDMA Read Request Header immediately 1058 follows the DDP header. The RDMAP layer passes to the DDP layer an 1059 RDMAP Control Field. The following figure depicts the RDMA Read 1060 Request Header that MUST be used for all RDMA Read Request 1061 Messages: 1063 0 1 2 3 1064 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1065 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1066 | Data Sink STag (SinkSTag) | 1067 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1068 | | 1069 + Data Sink Tagged Offset (SinkTO) + 1070 | | 1071 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1072 | RDMA Read Message Size (RDMARDSZ) | 1073 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 | Data Source STag (SrcSTag) | 1075 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1076 | | 1077 + Data Source Tagged Offset (SrcTO) + 1078 | | 1079 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1080 Figure 6 RDMA Read Request Header Format 1082 Data Sink Steering Tag: 32 bits. 1084 The Data Sink Steering Tag identifies the Data Sink's Tagged 1085 Buffer. This field MUST be copied, without interpretation, 1086 from the RDMA Read Request into the corresponding RDMA Read 1087 Response and allows the Data Sink to place the returning 1088 data. The STag is associated with the RDMAP Stream through a 1089 mechanism that is outside the scope of the RDMAP 1090 specification (see Section 10.3 Other Security 1091 Considerations). 1093 Data Sink Tagged Offset: 64 bits. 1095 The Data Sink Tagged Offset specifies the starting offset, in 1096 octets, from the base of the Data Sink's Tagged Buffer, where 1097 the data is to be written by the Data Source. This field is 1098 copied from the RDMA Read Request into the corresponding RDMA 1099 Read Response and allows the Data Sink to place the returning 1100 data. The Data Sink Tagged Offset MAY start at an arbitrary 1101 offset. 1103 The Data Sink STag and Data Sink Tagged Offset fields 1104 describe the buffer to which the RDMA Read data is written. 1106 Note: the DDP Layer protects against a wrap of the Data Sink 1107 Tagged Offset. 1109 RDMA Read Message Size: 32 bits. 1111 The RDMA Read Message Size is the amount of data, in octets, 1112 read from the Data Source. A single RDMA Read Request Message 1113 can retrieve from 0 to 2^32-1 data octets from the Data 1114 Source. 1116 Data Source Steering Tag: 32 bits. 1118 The Data Source Steering Tag identifies the Data Source's 1119 Tagged Buffer. The STag is associated with the RDMAP Stream 1120 through a mechanism that is outside the scope of the RDMAP 1121 specification (see Section 10.3 Other Security 1122 Considerations). 1124 Data Source Tagged Offset: 64 bits. 1126 The Tagged Offset specifies the starting offset, in octets, 1127 that is to be read from the Data Source's Tagged Buffer. The 1128 Data Source Tagged Offset MAY start at an arbitrary offset. 1130 The Data Source STag and Data Source Tagged Offset fields 1131 describe the buffer from which the RDMA Read data is read. 1133 See Section 9.2 Errors Detected at the Remote Peer on Incoming 1134 RDMA Messages for a description of error checking required upon 1135 processing of an RDMA Read Request at the Data Source. 1137 6.5 RDMA Read Response Header 1139 The RDMA Read Response Message does not include an RDMAP header. 1140 The RDMAP layer passes to the DDP layer an RDMAP Control Field. 1141 The RDMA Read Response Message is fully described by the DDP 1142 Headers of the DDP Segments associated with the Message. 1144 See Section 12 Appendix for a description of the DDP Segment 1145 format associated with RDMA Read Response Messages. 1147 6.6 Send Header and Send with Solicited Event Header 1149 The Send and Send with Solicited Event Message do not include an 1150 RDMAP header. The RDMAP layer passes to the DDP layer an RDMAP 1151 Control Field. The Send and Send with Solicited Event Message are 1152 fully described by the DDP Headers of the DDP Segments associated 1153 with the Message. 1155 See Section 12 Appendix for a description of the DDP Segment 1156 format associated with Send and Send with Solicited Event 1157 Messages. 1159 6.7 Send with Invalidate Header and Send with SE and Invalidate 1160 Header 1162 The Send with Invalidate and Send with Solicited Event and 1163 Invalidate Message do not include an RDMAP header. The RDMAP layer 1164 passes to the DDP layer an RDMAP Control Field and the Invalidate 1165 STag field (see section 6.1 RDMAP Control and Invalidate STag 1166 Field). The Send with Invalidate and Send with Solicited Event and 1167 Invalidate Message are fully described by the DDP Headers of the 1168 DDP Segments associated with the Message. 1170 See Section 12 Appendix for a description of the DDP Segment 1171 format associated with Send and Send with Solicited Event 1172 Messages. 1174 6.8 Terminate Header 1176 The Terminate Message carries a Terminate Header that contains 1177 additional information associated with the cause of the Terminate. 1178 The Terminate Header immediately follows the DDP header. The RDMAP 1179 layer passes to the DDP layer an RDMAP Control Field. The 1180 following figure depicts a Terminate Header that MUST be used for 1181 the Terminate Message: 1183 0 1 2 3 1184 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1185 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1186 | Terminate Control | Reserved | 1187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1188 | DDP Segment Length (if any) | | 1189 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 1190 | | 1191 // // 1192 | Terminated DDP Header (if any) | 1193 + + 1194 | | 1195 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1196 | | 1197 // // 1198 | Terminated RDMA Header (if any) | 1199 + + 1200 | | 1201 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1202 Figure 7 Terminate Header Format 1204 Terminate Control: 19 bits. 1206 The Terminate Control field MUST have the format defined in 1207 Figure 8 Terminate Control Field. 1209 0 1 2 3 1210 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1211 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1212 | Layer | EType | Error Code |HdrCt| 1213 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1214 Figure 8 Terminate Control Field 1216 * Figure 9 Terminate Control Field Values defines the valid 1217 values that MUST be used for this field. 1219 * Layer: 4 bits. 1221 Identifies the layer that encountered the error. 1223 * EType (RDMA Error Type): 4 bits. 1225 Identifies the type of error that caused the 1226 Terminate. When the error is detected at the RDMAP 1227 Layer, the RDMAP Layer inserts the Error Type into 1228 this field. When the error is detected at a LLP layer, 1229 a LLP layer creates the Error Type and the DDP layer 1230 passes it up to the RDMAP Layer, and the RDMAP Layer 1231 inserts it into this field. 1233 * Error Code: 8 bits. 1235 This field identifies the specific error that caused 1236 the Terminate. When the error is detected at the RDMAP 1237 Layer, the RDMAP Layer creates the Error Code. When 1238 the error is detected at a LLP layer, a LLP layer 1239 creates the Error Code and the DDP layer passes it up 1240 to the RDMAP Layer, and the RDMAP Layer inserts it 1241 into this field. 1243 * HdrCt: 3 bits. 1245 Header control bits: 1247 * M: bit 16. DDP Segment Length valid. See Figure 10 1248 for when this bit SHOULD be set. 1250 * D: bit 17. DDP Header Included. See Figure 10 for 1251 when this bit SHOULD be set. 1253 * R: bit 18. RDMAP Header Included. See Figure 10 1254 for when this bit SHOULD be set. 1256 -------+----------+-------+-------------+------+-------------------- 1257 Layer | Layer | Error | Error Type | Error| Error Code Name 1258 | Name | Type | Name | Code | 1259 -------+----------+-------+-------------+------+-------------------- 1260 | | 0000b | Local | None | None 1261 | | | Catastrophic| | 1262 | | | Error | | 1263 | +-------+-------------+------+-------------------- 1264 | | | | 00X | Invalid STag 1265 | | | +------+-------------------- 1266 | | | | 01X | Base or bounds 1267 | | | | | violation 1268 | | | Remote +------+-------------------- 1269 | | 0001b | Protection | 02X | Access rights 1270 | | | Error | | violation 1271 | | | +------+-------------------- 1272 0000b | RDMA | | | 03X | STag not associated 1273 | | | | | with RDMAP Stream 1274 | | | +------+-------------------- 1275 | | | | 04X | TO wrap 1276 | | | +------+-------------------- 1277 | | | | 09X | STag cannot be 1278 | | | | | Invalidated 1279 | | | +------+-------------------- 1280 | | | | FFX | Unspecified Error 1281 | +-------+-------------+------+-------------------- 1282 | | | | 05X | Invalid RDMAP 1283 | | | | | version 1284 | | | +------+-------------------- 1285 | | | | 06X | Unexpected OpCode 1286 | | | Remote +------+-------------------- 1287 | | 0010b | Operation | 07X | Catastrophic error, 1288 | | | Error | | localized to RDMAP 1289 | | | | | Stream 1290 | | | +------+-------------------- 1291 | | | | 08X | Catastrophic error, 1292 | | | | | global 1293 | | | +------+-------------------- 1294 | | | | 09X | STag cannot be 1295 | | | | | Invalidated 1296 | | | +------+-------------------- 1297 | | | | FFX | Unspecified Error 1298 -------+----------+-------+-------------+------+-------------------- 1299 0001b | DDP | See DDP Specification [DDP] for a description of 1300 | | the values and names. 1301 -------+----------+-------+----------------------------------------- 1302 0010b | LLP | For MPA, see MPA Specification [MPA] for a 1303 | (eg MPA) | description of the values and names. 1304 -------+----------+-------+----------------------------------------- 1305 Figure 9 Terminate Control Field Values 1307 Reserved: 8 bits. This field MUST be set to zero on transmit, 1308 ignored on receive. 1310 DDP Segment Length: 16 bits 1312 The length handed up by the DDP Layer when the error was 1313 detected. It MUST be valid if the M bit is set. It MUST be 1314 present when the D bit is set. 1316 Terminated DDP Header: 112 bits for Tagged Messages and 144 bits 1317 for Untagged Messages. 1319 The DDP Header of the incoming Message that is associated 1320 with the Terminate. The DDP Header is not present if the 1321 Terminate Error Type is a Local Catastrophic Error. It MUST 1322 be present if the D bit is set. 1324 Terminated RDMA Header: 224 bits. 1326 The Terminated RDMA Header is only sent back if the terminate 1327 is associated with an RDMA Read Request Message. It MUST be 1328 present if the R bit is set. 1330 If the terminate occurs before the first RDMA Read Request 1331 byte is processed, the original RDMA Read Request Header is 1332 sent back. 1334 If the terminate occurs after the first RDMA Read Request 1335 byte is processed, the RDMA Read Request Header is updated to 1336 reflect the current location of the RDMA Read operation that 1337 is in process: 1339 * Data Sink STag = Data Sink STag originally sent in the 1340 RDMA Read Request. 1342 * Data Sink Tagged Offset = Current offset into the Data 1343 Sink Tagged Buffer. For example if the RDMA Read 1344 Request was terminated after 2048 octets were sent, 1345 then the Data Sink Tagged Offset = the original Data 1346 Sink Tagged Offset + 2048. 1348 * Data Message size = Number of bytes left to transfer. 1350 * Data Source STag = Data Source STag in the RDMA Read 1351 Request. 1353 * Data Source Tagged Offset = Current offset into the 1354 Data Source Tagged Buffer. For example if the RDMA 1355 Read Request was terminated after 2048 octets were 1356 sent, then the Data Source Tagged Offset = the 1357 original Data Source Tagged Offset + 2048. 1359 Note: if a given LLP does not define any termination codes for the 1360 RDMAP Termination message to use, then none would be used for that 1361 LLP. 1363 Figure 10 Error Type to RDMA Message Mapping maps layer name and 1364 error types to each RDMA Message type: 1366 ---------+-------------+------------+------------+----------------- 1367 Layer | Error Type | Terminate | Terminate | What type of 1368 Name | Name | Includes | Includes | RDMA Message can 1369 | | DDP Header | RDMA Header| cause the error 1370 | | and DDP | | 1371 | | Segment | | 1372 | | Length | | 1373 ---------+-------------+------------+------------+----------------- 1374 | Local | No | No | Any 1375 | Catastrophic| | | 1376 | Error | | | 1377 +-------------+------------+------------+----------------- 1378 | Remote | Yes, if | Yes | Only RDMA Read 1379 RDMA | Protection | possible | | Request, Send 1380 | Error | | | with Invalidate, 1381 | | | | and Send with SE 1382 | | | | and Invalidate 1383 +-------------+------------+------------+----------------- 1384 | Remote | Yes, if | No | Any 1385 | Operation | possible | | 1386 | Error | | | 1387 ---------+-------------+------------+------------+----------------- 1388 DDP | See DDP Spec| Yes | No | Any 1389 | [DDP] | | | 1390 ---------+-------------+------------+------------+----------------- 1391 LLP | See LLP Spec| No | No | Any 1392 | [e.g. MPA] | | | 1393 Figure 10 Error Type to RDMA Message Mapping 1395 7 Data Transfer 1397 7.1 RDMA Write Message 1399 An RDMA Write is used by the Data Source to transfer data to a 1400 previously Advertised Tagged Buffer at the Data Sink. The RDMA 1401 Write Message has the following semantics: 1403 * AN RDMA Write Message MUST reference a Tagged Buffer. That is, 1404 the Data Source RDMAP Layer MUST request that the DDP layer 1405 mark the Message as Tagged. 1407 * A valid RDMA Write Message MUST NOT be delivered to the Data 1408 Sink's ULP (i.e. it is placed by the DDP layer). 1410 * At the Remote Peer, when an invalid RDMA Write Message is 1411 delivered to the Remote Peer's RDMAP Layer, an error is 1412 surfaced (see section 9.1 RDMAP Error Surfacing). 1414 * The Tagged Offset of a Tagged Buffer MAY start at a non-zero 1415 value. 1417 * AN RDMA Write Message MAY target all or part of a previously 1418 Advertised buffer. 1420 * The RDMAP does not define how the buffer(s) used by an outbound 1421 RDMA Write is defined and how it is addressed. For example, an 1422 implementation of RDMA may choose to allow a gather-list of 1423 non-contiguous data blocks to be the source of an RDMA Write. 1424 In this case, the data blocks would be combined by the Data 1425 Source and sent as a single RDMA Write Message to the Data 1426 Sink. 1428 * The Data Source RDMAP Layer MUST issue RDMA Write Messages to 1429 the DDP layer in the order they were submitted by the ULP. 1431 * At the Data Source, a subsequent Send (Send with Invalidate, 1432 Send with Solicited Event, or Send with Solicited Event and 1433 Invalidate) Message MAY be used to signal Delivery of previous 1434 RDMA Write Messages to the Data Sink, if desired by the ULP. 1436 * If the Local Peer wishes to write to multiple Tagged Buffers on 1437 the Remote Peer, the Local Peer MUST use multiple RDMA Write 1438 Messages. That is, a single RDMA Write Message can only write 1439 to one remote Tagged Buffer. 1441 * The Data Source MAY issue a zero length RDMA Write Message. 1443 7.2 RDMA Read Operation 1445 The RDMA Read operation MUST consist of a single RDMA Read Request 1446 Message and a single RDMA Read Response Message. 1448 7.2.1 RDMA Read Request Message 1450 An RDMA Read Request is used by the Data Sink to transfer data 1451 from a previously Advertised Tagged Buffer at the Data Source to a 1452 Tagged Buffer at the Data Sink. The RDMA Read Request Message has 1453 the following semantics: 1455 * AN RDMA Read Request Message MUST reference an Untagged Buffer. 1456 That is, the Local Peer's RDMAP Layer MUST request that the DDP 1457 mark the Message as Untagged. 1459 * One RDMA Read Request Message MUST consume one Untagged Buffer. 1461 * The Remote Peer's RDMAP Layer MUST process an RDMA Read Request 1462 Message. A valid RDMA Read Request Message MUST NOT be 1463 delivered to the Data Sink's ULP (i.e. it is processed by the 1464 RDMAP layer). 1466 * At the Remote Peer, when an invalid RDMA Read Request Message 1467 is delivered to the Remote Peer's RDMAP Layer, an error is 1468 surfaced (see section 9.1 RDMAP Error Surfacing). 1470 * AN RDMA Read Request Message MUST reference the RDMA Read 1471 Request Queue. That is, the Local Peer's RDMAP Layer MUST 1472 request that the DDP layer set the Queue Number field to one. 1474 * The Local Peer MUST pass to the DDP Layer RDMA Read Request 1475 Messages in the order they were submitted by the ULP. 1477 * The Remote Peer MUST process the RDMA Read Request Messages in 1478 the order they were sent. 1480 * If the Local Peer wishes to read from multiple Tagged Buffers 1481 on the Remote Peer, the Local Peer MUST use multiple RDMA Read 1482 Request Messages. That is, a single RDMA Read Request Message 1483 MUST only read from one remote Tagged Buffer. 1485 * AN RDMA Read Request Message MAY target all or part of a 1486 previously Advertised buffer. 1488 * If the Data Source receives a valid RDMA Read Request Message 1489 it MUST respond with a valid RDMA Read Response Message. 1491 * The Data Sink MAY issue a zero length RDMA Read Request 1492 Message, by setting the RDMA Read Message Size field to zero in 1493 the RDMA Read Request Header. 1495 * If the Data Source receives a non-zero length RDMA Read Message 1496 Size, the Data Source RDMAP MUST validate the Data Source STag 1497 and Data Source Tagged Offset contained in the RDMA Read 1498 Request Header. 1500 * If the Data Source receives an RDMA Read Request Header with 1501 the RDMA Read Message Size set to zero, the Data Source RDMAP: 1503 * MUST NOT validate the Data Source STag and Data Source 1504 Tagged Offset contained in the RDMA Read Request Header, 1505 and 1507 * MUST respond with a zero length RDMA Read Response 1508 Message. 1510 7.2.2 RDMA Read Response Message 1512 The RDMA Read Response Message uses the DDP Tagged Buffer Model to 1513 Deliver the contents of a previously requested Data Source Tagged 1514 Buffer to the Data Sink, without any involvement from the ULP at 1515 the Remote Peer. The RDMA Read Response Message has the following 1516 semantics: 1518 * The RDMA Read Response Message for the associated RDMA Read 1519 Request Message travels in the opposite direction. 1521 * An RDMA Read Response Message MUST reference a Tagged Buffer. 1522 That is, the Data Source RDMAP Layer MUST request that the DDP 1523 mark the Message as Tagged. 1525 * The Data Source MUST ensure that a sufficient number of 1526 Untagged Buffers are available on the RDMA Read Request Queue 1527 (Queue with DDP Queue Number 1) to support the maximum number 1528 of RDMA Read Requests negotiated by the ULP. 1530 * The RDMAP Layer MUST Deliver the RDMA Read Response Message to 1531 the ULP. 1533 * At the Remote Peer, when an invalid RDMA Read Response Message 1534 is delivered to the Remote Peer's RDMAP Layer, an error is 1535 surfaced (see section 9.1 RDMAP Error Surfacing). 1537 * The Tagged Offset of a Tagged Buffer MAY start at a non-zero 1538 value. 1540 * The Data Source RDMAP Layer MUST pass RDMA Read Response 1541 Messages to the DDP layer in the order that the RDMA Read 1542 Request Messages were received by the RDMAP Layer at the Data 1543 Source. 1545 * The Data Sink MAY validate that the STag, Tagged Offset, and 1546 length of the RDMA Read Response Message are the same as the 1547 STag, Tagged Offset, and length included in the corresponding 1548 RDMA Read Request Message. 1550 * A single RDMA Read Response Message MUST write to one remote 1551 Tagged Buffer. If the Data Sink wishes to Read multiple Tagged 1552 Buffers, the Data Sink can use multiple RDMA Read Request 1553 Messages. 1555 7.3 Send Message Type 1557 The Send Message Type uses the DDP Untagged Buffer Model to 1558 transfer data from the Data Source into an Untagged Buffer at the 1559 Data Sink. 1561 * A Send Message Type MUST reference an Untagged Buffer. That is, 1562 the Local Peer's RDMAP Layer MUST request that the DDP layer 1563 mark the Message as Untagged. 1565 * One Send Message Type MUST consume one Untagged Buffer. 1567 * The ULP Message sent using a Send Message Type MAY be less 1568 than or equal to the size of the consumed Untagged Buffer. 1569 The RDMAP Layer communicates to the ULP the size of the 1570 data written into the Untagged Buffer. 1572 * If the ULP Message sent via Send Message Type is larger 1573 than the Data Sink's Untagged Buffer, it is an error (see 1574 section 9.1 RDMAP Error Surfacing). 1576 * At the Remote Peer, the Send Message Type MUST be Delivered to 1577 the Remote Peer's ULP in the order they were sent. 1579 * After the Send with Solicited Event or Send with Solicited 1580 Event and Invalidate Message is Delivered to the ULP, the RDMAP 1581 MAY generate an Event, if the Data Sink is configured to 1582 generate such an Event. 1584 * At the Remote Peer, when an invalid Send Message Type is 1585 Delivered to the Remote Peer's RDMAP Layer, an error is 1586 surfaced (see section 9.1 RDMAP Error Surfacing). 1588 * The RDMAP does not define how the buffer(s) used by an outbound 1589 Send Message Type is defined and how it is addressed. For 1590 example, an implementation of RDMA may choose to allow a 1591 gather-list of non-contiguous data blocks to be the source of a 1592 Send Message Type. In this case, the data blocks would be 1593 combined by the Data Source and sent as a single Send Message 1594 Type to the Data Sink. 1596 * For a Send Message Type, the Local Peer's RDMAP Layer MUST 1597 request that the DDP layer set the Queue Number field to zero. 1599 * The Local Peer MUST issue Send Message Type Messages in the 1600 order they were submitted by the ULP. 1602 * The Data Source MAY pass a zero length Send Message Type. A 1603 zero length Send Message Type MUST consume an Untagged Buffer 1604 at the Data Sink. A Send with Invalidate or Send with Solicited 1605 Event and Invalidate Message MUST reference an STag. That is, 1606 the Local Peer's RDMAP Layer MUST pass the RDMA control field 1607 and the STag that will be Invalidated to the DDP layer. 1609 * When the Send with Invalidate and Send with Solicited Event and 1610 Invalidate Message are Delivered to the Remote Peer's RDMAP 1611 Layer, the RDMAP Layer MUST: 1613 * Verify the STag that is associated with the RDMAP Stream; 1614 and 1616 * Invalidate the STag if it is associated with the RDMAP 1617 Stream; or Issue a Terminate Message with the STag Cannot 1618 be Invalidated Terminate Error Code, if the STag is not 1619 associated with the RDMAP Stream. 1621 7.4 Terminate Message 1623 The Terminate Message uses the DDP Untagged Buffer Model to 1624 transfer error related information from the Data Source into an 1625 Untagged Buffer at the Data Sink and then ceases all further 1626 communications on the underlying DDP Stream. The Terminate Message 1627 has the following semantics: 1629 * A Terminate Message MUST reference an Untagged Buffer. That is, 1630 the Local Peer's RDMAP Layer MUST request that the DDP layer 1631 mark the Message as Untagged. 1633 * A Terminate Message references the Terminate Queue. That is, 1634 the Local Peer's RDMAP Layer MUST request that the DDP layer 1635 set the Queue Number field to two. 1637 * One Terminate Message MUST consume one Untagged Buffer. 1639 * On a single RDMAP Stream, the RDMAP layer MUST guarantee 1640 placement of a single Terminate Message. 1642 * A Terminate Message MUST be Delivered to the Remote Peer's 1643 RDMAP Layer. The RDMAP Layer MUST Deliver the Terminate Message 1644 to the ULP. 1646 * At the Remote Peer, when an invalid Terminate Message is 1647 delivered to the Remote Peer's RDMAP Layer, an error is 1648 surfaced (see section 9.1 RDMAP Error Surfacing). 1650 * The RDMAP Layer Completes in error all ULP Operations that have 1651 not been provided to the DDP layer. 1653 * After sending a Terminate Message on an RDMAP Stream, the Local 1654 Peer MUST NOT send any more Messages on that specific RDMAP 1655 Stream. 1657 * After receiving a Terminate Message on an RDMAP Stream, the 1658 Remote Peer MAY stop sending Messages on that specific RDMAP 1659 Stream. 1661 7.5 Ordering and Completions 1663 It is important to understand the difference between Placement and 1664 Delivery ordering since RDMAP provides quite different semantics 1665 for the two. 1667 Note that many current protocols, both as used in the Internet and 1668 elsewhere, assume that data is both Placed and Delivered in order. 1669 This allowed applications to take a variety of shortcuts by taking 1670 advantage of this fact. For RDMAP, many of these shortcuts are no 1671 longer safe to use, and could cause application failure. 1673 The following rules apply to implementations of the RDMAP 1674 protocol. Note, in these rules Send includes Send, Send with 1675 Invalidate, Send with Solicited Event, and Send with Solicited 1676 Event and Invalidate: 1678 1. RDMAP does not provide ordering among Messages on different 1679 RDMAP Streams. 1681 2. RDMAP does not provide ordering between operations that are 1682 generated from the two ends of an RDMAP Stream. 1684 3. RDMA Messages that use Tagged and Untagged Buffers MAY be 1685 Placed in any order. If an application uses overlapping 1686 buffers (points different Messages or portions of a single 1687 Message at the same buffer), then it is possible that the last 1688 incoming write to the Data Sink buffer will not be the last 1689 outgoing data sent from the Data Source. 1691 4. For a Send operation, the contents of an Untagged Buffer at 1692 the Data Sink MAY be indeterminate until the Send is Delivered 1693 to the ULP at the Data Sink. 1695 5. For an RDMA Write operation, the contents of the Tagged Buffer 1696 at the Data Sink MAY be indeterminate until a subsequent Send 1697 is Delivered to the ULP at the Data Sink. 1699 6. For an RDMA Read operation, the contents of the Tagged Buffer 1700 at the Data Sink MAY be indeterminate until the RDMA Read 1701 Response Message has been Delivered at the Local Peer. 1703 Statements 4, 5, and 6 imply "no peeking" at the data to see 1704 if it is done. It is possible for some data to arrive before 1705 logically earlier data does, and peeking may cause 1706 unpredictable application failure 1708 7. If the ULP or Application modifies the contents of Tagged or 1709 Untagged Buffers being modified by an RDMA Operation while the 1710 RDMAP is processing the RDMA Operation, the state of the 1711 Buffers is indeterminate. 1713 8. If the ULP or Application modifies the contents of Tagged or 1714 Untagged Buffers read by an RDMA Operation while the RDMAP is 1715 processing the RDMA Operation, the results of the read are 1716 indeterminate. 1718 9. The Completion of an RDMA Write or Send Operation at the Local 1719 Peer does not guarantee that the ULP Message has yet reached 1720 the Remote Peer ULP Buffer or been examined by the Remote ULP. 1722 10. Send Messages MUST be Delivered to the ULP at the Remote Peer 1723 after they are Delivered to RDMAP by DDP and in the order that 1724 the they were Delivered to RDMAP. 1726 Note that DDP ordering rules ensure that this will be the same 1727 order that they were submitted at the Local Peer and that any 1728 prior RDMA Writes have been submitted for ordered Placement at 1729 the Remote Peer. This means that when the ULP sees the 1730 Delivery of the Send, the memory buffers targeted by any 1731 preceding RDMA Writes and Sends are available to be accessed 1732 locally or remotely as authorized. If the ULP overlaps its 1733 buffers for different operations, the data from the RDMA Write 1734 or Send may be overwritten by subsequent RDMA Operations 1735 before the ULP receives and processes the Delivery. 1737 11. RDMA Read Response Messages MUST be Delivered to the ULP at 1738 the Remote Peer after they are Delivered to RDMAP by DDP and 1739 in the order that the they were Delivered to RDMAP. 1741 DDP ordering rules ensure that this will be the same order 1742 that they were submitted at the Local Peer. This means that 1743 when the ULP sees the Delivery of the RDMA Read Response, the 1744 memory buffers targeted by the RDMA Read Response are 1745 available to be accessed locally or remotely as authorized. If 1746 the ULP overlaps its buffers for different operations, the 1747 data from the RDMA Read Response may be overwritten by 1748 subsequent RDMA Operations before the ULP receives and 1749 processes the Delivery. 1751 12. RDMA Read Request Messages, including zero-length RDMA Read 1752 Requests, MUST NOT start processing at the Remote Peer until 1753 they have been Delivered to RDMAP by DDP. 1755 Note: the ULP is assured that data written can be read back. 1756 For example, if an RDMA Read Request is issued by the local 1757 peer, targeting the same ULP Buffer as a preceding Send or 1758 RDMA Write (in the same direction as the RDMA Read Request), 1759 and there are no other sources of update for the ULP Buffer, 1760 then the remote peer will send back the data written by the 1761 Send or RDMA Write. That is, for this example the ULP Buffer: 1762 is Advertised for use on a series of RDMA Messages, is only 1763 valid on the RDMAP Stream for which it is advertised, and is 1764 not locally updated while the series of RDMAP Messages are 1765 performed. For this example, order rule (12) assures that 1766 subsequent local or remote accesses to the ULP Buffer contain 1767 the data written by the Send or RDMA Write. 1769 RDMA Read Response Messages MAY be generated at the Remote 1770 Peer after subsequent RDMA Write Messages or Send Messages 1771 have been Placed or Delivered. Therefore, when an application 1772 does an RDMA Read Request followed by an RDMA Write (or Send) 1773 to the same buffer, it may get the data from the later RDMA 1774 Write (or Send) in the RDMA Read Response Message, even though 1775 the operations completed in order at the Local Peer. If this 1776 behavior is not desired, the Local Peer ULP must Fence the 1777 later RDMA write (or Send) by withholding the RDMA Write 1778 Message until all outstanding RDMA Read Responses have been 1779 Delivered. 1781 13. The RDMAP Layer MUST submit RDMA Messages to the DDP layer in 1782 the order the RDMA Operations are submitted to the RDMAP Layer 1783 by the ULP. 1785 14. A Send or RDMA Write Message MUST NOT be considered Complete 1786 at the Local Peer (Data Source) until it has been successfully 1787 completed at the DDP layer. 1789 15. RDMA Operations MUST be Completed at the Local Peer in the 1790 order that they were submitted by the ULP. 1792 16. At the Data Sink, an incoming Send Message MUST be Delivered 1793 to the ULP only after the DDP Message has been Delivered to 1794 the RDMAP Layer by the DDP layer. 1796 17. RDMA Read Response Message processing at the Remote Peer 1797 (reading the specified Tagged Buffer) MUST be started only 1798 after the RDMA Read Request Message has been Delivered by the 1799 DDP layer (thus all previous RDMA Messages have been properly 1800 submitted for ordered Placement). 1802 18. Send Messages MAY be Completed at the Remote Peer (Data Sink) 1803 before prior incoming RDMA Read Request Messages have 1804 completed their response processing. 1806 19. An RDMA Read operation MUST NOT be Completed at the Local Peer 1807 until the DDP layer Delivers the associated incoming RDMA Read 1808 Response Message. 1810 20. If more than one outstanding RDMA Read Request Message is 1811 supported by both peers, the RDMA Read Response Messages MUST 1812 be submitted to the DDP layer on the Remote Peer in the order 1813 the RDMA Read Request Messages were Delivered by DDP, but the 1814 actual read of the buffer contents MAY take place in any order 1815 at the Remote Peer. 1817 This simplifies Local Peer Completion processing for RDMA 1818 Reads in that a Delivered RDMA Read Response MUST be 1819 sufficient to Complete the RDMA Read Operation. 1821 8 RDMAP Stream Management 1823 RDMAP Stream management consists of RDMAP Stream Initialization 1824 and RDMAP Stream Termination. 1826 8.1 Stream Initialization 1828 RDMAP Stream initialization occurs after the LLP Stream has been 1829 created (e.g. for DDP/MPA over TCP the first TCP Segment after the 1830 SYN, SYN/ACK exchange). The ULP is responsible for transitioning 1831 the LLP Stream into RDMA enabled mode. The switch to RDMA mode 1832 typically occurs sometime after LLP Stream. Once in RDMA enabled 1833 mode, an implementation MUST send only RDMA Messages across the 1834 transport Stream until the RDMAP Stream is torn down. 1836 For each direction of an RDMAP Stream: 1838 * For a given RDMAP Stream, the number of outstanding RDMA Read 1839 Requests is limited per RDMAP Stream direction. 1841 * It is the ULP's responsibility to set the maximum number of 1842 outstanding, inbound RDMA Read Requests per RDMAP Stream 1843 direction. 1845 * The RDMAP Layer MUST provide the maximum number of outstanding, 1846 inbound RDMA Read Requests per RDMAP Stream direction that were 1847 negotiated between the ULP and the Local Peer's RDMAP Layer. 1848 The negotiation mechanism is outside the scope of this 1849 specification. 1851 * It is the ULP's responsibility to set the maximum number of 1852 outstanding, outbound RDMA Read Requests per RDMAP Stream 1853 direction. 1855 * The RDMAP Layer MUST provide the maximum number of outstanding, 1856 outbound RDMA Read Requests for the RDMAP Stream direction that 1857 were negotiated between the ULP and the Local Peer's RDMAP 1858 Layer. The negotiation mechanism is outside the scope of this 1859 specification. 1861 * The Local Peer's ULP is responsible for negotiating with the 1862 Remote Peer's ULP the maximum number of outstanding RDMA Read 1863 Requests for the RDMAP Stream direction. It is recommended that 1864 the ULP set the maximum number of outstanding, inbound RDMA 1865 Read Requests equal to the maximum number of outstanding, 1866 outbound RDMA Read Requests for a given RDMAP Stream direction. 1868 * For outbound RDMA Read Requests, the RDMAP Layer MUST NOT 1869 exceed the maximum number of outstanding, outbound RDMA Read 1870 Requests that were negotiated between the ULP and the Local 1871 Peer's RDMAP Layer. 1873 * For inbound RDMA Read Requests, the RDMAP Layer MUST NOT exceed 1874 the maximum number of outstanding, inbound RDMA Read Requests 1875 that were negotiated between the ULP and the Local Peer's RDMAP 1876 Layer. 1878 8.2 Stream Teardown 1880 There are three methods for terminating an RDMAP Stream: ULP 1881 Graceful Termination, RDMAP Abortive Termination, and LLP Abortive 1882 Termination. 1884 The ULP is responsible for performing ULP Graceful Termination. 1885 After a ULP Graceful Termination, either side of the Stream can 1886 initiate LLP Graceful Termination, using the graceful termination 1887 mechanism provided by the LLP. 1889 RDMAP Abortive Termination allows the RDMAP to issue a Terminate 1890 Message describing the reason the RDMAP Stream was terminated. The 1891 next section (8.2.1 RDMAP Abortive Termination) describes the 1892 RDMAP Abortive Termination in detail. 1894 LLP results due to a LLP error and causes the RDMAP Stream to be 1895 torn down midstream, without an RDMAP Terminate Message. While 1896 this last method is highly undesirable, it is possible and the ULP 1897 should take this into consideration. 1899 8.2.1 RDMAP Abortive Termination 1901 RDMAP defines a Terminate operation that SHOULD be invoked when 1902 either an RDMAP error is encountered or a LLP error is surfaced to 1903 the RDMAP layer by the LLP. 1905 It is not always possible to send the Terminate Message. For 1906 example, certain LLP errors may occur that cause the LLP Stream to 1907 be torn down before a) RDMAP is aware of the error, b) before 1908 RDMAP is able to send the Terminate Message, or c) after RDMAP has 1909 posted the Terminate Message to the LLP, but it has not yet been 1910 transmitted by the LLP. 1912 Note that an RDMAP Abortive Termination may entail loss of data. 1913 In general, when a Terminate Message is received it is impossible 1914 to tell for sure what unacknowledged RDMA Messages were Completed 1915 successfully at the Remote Peer. Thus the state of all outstanding 1916 RDMA Messages is indeterminate and the Messages SHOULD be 1917 considered Completed in error. 1919 When a peer sends or receives a Terminate Message, it MAY 1920 immediately teardown the LLP Stream. The peer SHOULD perform a 1921 graceful LLP teardown to ensure the Terminate Message is 1922 successfully Delivered. 1924 See section 6.8 Terminate Header for a description of the 1925 Terminate Message and its contents. See section 7.4 Terminate 1926 Message for a description of the Terminate Message semantics. 1928 9 RDMAP Error Management 1930 The RDMAP protocol does not have RDMAP or DDP layer error recovery 1931 operations built in. If everything is working, the LLP guarantees 1932 will ensure that the Messages are arriving at the destination. 1934 If errors are detected at the RDMAP or DDP layer, then the RDMAP, 1935 DDP and LLP Streams are Abortively Terminated (see section 6.8 1936 Terminate Header on page 32). 1938 In general poor implementations or improper ULP programming causes 1939 the errors detected at the RDMAP and DDP layers. In these cases, 1940 returning a diagnostic termination error Message and closing the 1941 RDMAP Stream is far simpler than attempting to maintain the RDMAP 1942 Stream, particularly when the cause of the error is not known. 1944 If an LLP does not support teardown of a Stream independent of 1945 other Streams and an RDMAP error results in the Termination of a 1946 specific Stream, then the LLP MUST label the Stream as an 1947 erroneous Stream and MUST NOT allow any further data transfer on 1948 that Stream after RDMAP requests the Stream to be torn down. 1950 For a specific LLP connection, when all Streams are either 1951 gracefully torn down or are labeled as erroneous Streams, the LLP 1952 connection MUST be torn down. 1954 Since errors are detected at the Remote Peer (possibly long) after 1955 RDMA Messages are passed to DDP and the LLP at the Local Peer and 1956 Completed, the sender cannot easily determine which of its 1957 Messages have been received. (RDMA Reads are an exception to this 1958 rule). 1960 For a list of errors returned to the Remote Peer as a result of an 1961 Abortive Termination, see section 6.8 Terminate Header on page 32. 1963 9.1 RDMAP Error Surfacing 1965 If an error occurs at the Local Peer, the RDMAP layer MUST attempt 1966 to inform the local ULP that the error has occurred. 1968 The Local Peer MUST send a Terminate Message for each of the 1969 following cases: 1971 1. For Errors detected while creating RDMA Write, Send, Send with 1972 Invalidate, Send with Solicited Event, Send with Solicited 1973 Event and Invalidate, or RDMA Read Requests, or other reasons 1974 not directly associated with an incoming Message, the 1975 Terminate Message and Error code are sent instead of the 1976 request. In this case, the Error Type and Error Code fields 1977 are included in the Terminate Message, but the Terminated DDP 1978 Header and Terminated RDMA Header fields are set to zero. 1980 2. For errors detected on an incoming RDMA Write, Send, Send with 1981 Invalidate, Send with Solicited Event, Send with Solicited 1982 Event and Invalidate, or Read Response Message (after the 1983 Message has been Delivered by DDP), the Terminate Message is 1984 sent at the earliest possible opportunity, preferably in the 1985 next outgoing RDMA Message. In this case, the Error Type, 1986 Error Code, ULP PDU Length, and Terminated DDP Header fields 1987 are included in the Terminate Message, but the Terminated RDMA 1988 Header field is set to zero. 1990 3. For errors detected on an incoming RDMA Read Request Message 1991 (after the Message has been Delivered by DDP), the Terminate 1992 Message is sent at the earliest possible opportunity, 1993 preferably in the next outgoing RDMA Message. In this case, 1994 the Error Type, Error Code, ULP PDU Length, Terminated DDP 1995 Header, and Terminated RDMA Header fields are included in the 1996 Terminate Message. 1998 4. If more than one error is detected on incoming RDMA Messages, 1999 before the Terminate Message can be sent, then the first RDMA 2000 Message (and its associated DDP Segment) that experienced an 2001 error MUST be captured by the Terminate Message in accordance 2002 with rules 2 and 3 above. 2004 9.2 Errors Detected at the Remote Peer on Incoming RDMA Messages 2006 On incoming RDMA Writes, RDMA Read Response, Sends, Send with 2007 Invalidate, Send with Solicited Event, Send with Solicited Event 2008 and Invalidate, and Terminate Messages, the following must be 2009 validated: 2011 1. The DDP Layer MUST validate all DDP Segment fields. 2013 2. The RDMA OpCode MUST be valid. 2015 3. The RDMA Version MUST be valid. 2017 Additionally, on incoming Send with Invalidate and Send with 2018 Solicited Event and Invalidate Messages, the following must 2019 also be validated: 2021 4. The Invalidate STag MUST be valid. 2023 5. The STag MUST be associated to this RDMAP Stream. 2025 On incoming RDMA Request Messages, the following must be 2026 validated: 2028 1. The DDP Layer MUST validate all Untagged DDP Segment fields. 2030 2. The RDMA OpCode MUST be valid. 2032 3. The RDMA Version MUST be valid. 2034 4. For non-zero length RDMA Read Request Messages: 2036 a. The Data Source STag MUST be valid. 2038 b. The Data Source STag MUST be associated to this RDMAP 2039 Stream. 2041 c. The Data Source Tagged Offset MUST fall in the range of 2042 legal offsets associated with the Data Source STag. 2044 d. The sum of the Data Source Tagged Offset and the RDMA Read 2045 Message Size MUST fall in the range of legal offsets 2046 associated with the Data Source STag. 2048 e. The sum of the Data Source Tagged Offset and the RDMA Read 2049 Message Size MUST NOT cause the Data Source Tagged Offset 2050 to wrap. 2052 10 Security Considerations 2054 This section discusses both protocol-specific considerations and 2055 the implications of using RDMAP with existing security mechanisms. 2056 A more detailed analysis of the security issues around the 2057 implementation and the use of the RDMAP can be found in [RDMASEC]. 2059 10.1 Protocol-specific Security Considerations 2061 The vulnerabilities of RDMAP to active third-party interference 2062 are no greater than any other protocol running over TCP. A third 2063 party, by injecting spoofed packets into the network that are 2064 Delivered to an RDMAP Data Sink, could launch a variety of attacks 2065 that exploit RDMAP-specific behavior. Since RDMAP directly or 2066 indirectly exposes memory addresses on the wire, the Placement 2067 information carried in each RDMA Message must be validated, 2068 including access rights and octet level granularity base and 2069 bounds check, before any data is Placed. For example, a third- 2070 party adversary could inject random packets that appear to be 2071 valid RDMA Messages and corrupt the memory on an RDMAP Data Sink. 2072 Since RDMAP is IP transport protocol independent, communication 2073 security mechanisms such as IPsec [IPSEC] or TLS [TLS] may be used 2074 to prevent such attacks. 2076 10.2 Using IPSec with RDMAP 2078 IPsec can be used to protect against the packet injection attacks 2079 outlined above. Because IPsec is designed to secure arbitrary IP 2080 packet streams, including streams where packets are lost, RDMAP 2081 can run on top of IPsec without any change. IPsec packets are 2082 processed (e.g., integrity checked and possibly decrypted) in the 2083 order they are received, and an RDMAP Data Sink will process the 2084 decrypted RDMA Messages contained in these packets in the same 2085 manner as RDMA Messages contained in unsecured IP packets. 2087 10.3 Other Security Considerations 2089 RDMAP has several mechanisms that deal with a number of attacks. 2090 These include, but are not limited to: 2092 1. Connection to/from an unauthorized or unauthenticated 2093 endpoint. 2095 2. Highjacking of an RDMAP Stream. 2097 3. Attempts to read or write from unauthorized memory regions. 2099 4. Injection of RDMA Messages within a Stream on a multi-user 2100 operating system by another application. 2102 RDMAP relies on the LLP to establish the LLP Stream over which 2103 RDMA Messages will be carried. RDMAP itself does nothing to 2104 authenticate the validity of the LLP Stream of either of the 2105 endpoints. It is the responsibility of the ULP to validate the 2106 LLP Stream. This is highly desirable due to the nature of RDMA 2107 and DDP. 2109 Hijacking of an RDMAP channel would require that the underlying 2110 LLP connection be hijacked. This would require knowledge of 2111 Advertised buffers in order to directly Place data into a user 2112 buffer and is therefore constrained by the same techniques 2113 mentioned to guard against attempts to read or write from 2114 unauthorized memory regions. 2116 RDMAP does not require a host to open its buffers to arbitrary 2117 attacks over the RDMAP Stream. It may access ULP memory only to 2118 the extent that the ULP has enabled and authorized it to do so. 2119 For a description of the STag access control model see [RDMASEC]. 2120 Specific security operations include: 2122 1. STags are only valid over the exact byte range established by 2123 the ULP. RDMAP MUST provide a mechanism for the ULP to 2124 establish and revoke the TO range associated with the ULP 2125 Buffer referenced by an STag. 2127 2. STags are only valid for the duration established by the ULP. 2128 The ULP may revoke them at any time, in accordance with its 2129 own upper layer protocol requirements. RDMAP MUST provide a 2130 mechanism for the ULP to establish and revoke STag validity. 2132 3. STags are only enabled for read and/or write access by 2133 explicit ULP action. RDMAP MUST provide a mechanism for the 2134 ULP to establish and revoke read, write, or read and write 2135 access to the ULP Buffer referenced by an STag. 2137 4. The implementation is free to choose the value of STags and is 2138 encouraged to sparsely populate them over the full range 2139 available. This is admittedly weak security protection against 2140 a deliberate attack, but does minimize the risk of accidental 2141 matches when an incorrect STag is used due to a ULP software 2142 error. 2144 5. RDMAP allows and encourages local interactions to restrict the 2145 usage of STags to specific Streams and/or user processes. 2146 RDMAP MUST provide a mechanism for associating a RDMAP Stream 2147 with a STag. 2149 RDMAP MAY choose several locally defined mechanisms for 2150 associating a RDMAP Stream and a STag. One such mechanism is 2151 to provide two association types: a protection domain 2152 association and an RDMAP Stream association. For a definition 2153 of protection domains and associated issues see [RDMASEC]. 2155 * Under the protection domain (PD) association, a unique 2156 protection domain identifier (PD ID) is created and used 2157 locally to associate the STag with a set of RDMAP Streams. 2158 The scope of the PD ID MUST include all of the RDMAP 2159 Streams associated with the PD ID. Under this mechanism, 2160 only RDMAP Streams that have the same PD ID as the STag 2161 are allowed to use the STag. 2163 For an incoming RDMA Read Request Message on an RDMAP 2164 Stream, if the PD ID associated with that RDMAP Stream is 2165 not the same as the PD ID associated with the Data Source 2166 STag, then no RDMA Read Response Message is returned and 2167 the RDMAP Stream MUST be Terminated with an Invalid STag 2168 error. For a Send with Invalidate or Send with SE and 2169 Invalidate on an RDMAP Stream, if the PD ID associated 2170 with that RDMAP Stream is not the same as the PD ID 2171 associated with the STag that is to be Invalidated, the 2172 Message is not delivered to the ULP and the RDMAP Stream 2173 MUST be Terminated with an STag cannot be Invalidated 2174 error. Note that the PD ID is locally defined, and cannot 2175 be directly manipulated by the Remote Peer. 2177 * Under RDMAP Stream association, a given RDMAP Stream is 2178 identified locally by a unique RDMAP Stream identifier 2179 (ID) and that RDMA Stream ID is associated with the STag 2180 and RDMAP Stream. 2182 For an incoming RDMA Read Request Message on an RDMAP 2183 Stream, if the RDMAP Stream ID associated with that RDMAP 2184 Stream is not the same as the RDMAP Stream ID associated 2185 with the Data Source STag, then no RDMA Read Response 2186 Message is returned and the RDMAP Stream MUST be 2187 Terminated with an Invalid STag error. Finally, for a Send 2188 with Invalidate or Send with SE and Invalidate on an RDMAP 2189 Stream, if the RDMAP Stream ID associated with that RDMAP 2190 Stream is not the same as the RDMAP Stream ID associated 2191 with the STag that is to be Invalidated, then the Message 2192 is not delivered to the ULP and the RDMAP Stream MUST be 2193 Terminated with an STag cannot be Invalidated error. Note 2194 that the RDMA Stream ID is locally defined, and cannot be 2195 directly manipulated by the Remote Peer. 2197 For an incoming RDMA Write or RDMA Read Response Message on an 2198 RDMAP Stream, the DDP layer MUST associate the STag targeted 2199 by the RDMA Write or RDMA Read Response Message (respectively) 2200 to the RDMAP Stream using one of the association mechanisms 2201 described above. If an STag targeted by the RDMA Write or 2202 RDMA Read Response Message Segment is not associated with the 2203 RDMAP Stream that received the Message Segment, DDP MUST 2204 surface an Invalid STag error to the RDMAP layer. The RDMAP 2205 layer MUST Terminate the RDMAP Stream if the DDP Layer 2206 surfaces an Invalid STag error. 2208 6. A ULP may only expose memory to remote access to the extent 2209 that it already had access to that memory itself. 2211 7. RDMAP provides operations to allow the holder of an STag to 2212 indicate when it has made its last usage of that STag. This 2213 enables automatic deregistration and/or scope reduction of 2214 STags as the implementation and ULP may see fit. Note, see 2215 [RDMASEC] for a description of the issues that arise when the 2216 ULP attempts to use a buffer that is still in use by RDMAP. 2218 8. If an STag is not valid on a connection, RDMAP provides a 2219 mechanism for terminating the RDMAP Stream (see section 6.8 2220 Terminate Header). 2222 9. An STag that is associated with an RDMAP Stream becomes 2223 invalid upon reception of a valid Send with Invalidate or Send 2224 with Solicited Event and Invalidate Message. RDMAP MUST 2225 invalidate the STag sent in a valid Send with Invalidate or 2226 Send with Solicited Event and Invalidate Message, before 2227 Completing the Send with Invalidate or Send with Solicited 2228 Event and Invalidate Message. 2230 Further, RDMAP encourages direct Placement of incoming payloads in 2231 user-mode ULP Buffers. This avoids the risks of prior solutions 2232 that relied upon exposing system buffers for incoming payloads. 2234 There is also a clean data Delivery hand-off between RDMAP and the 2235 ULP. This allows the ULP to implement additional security 2236 operations without restrictions or interference from RDMAP. 2238 In summary, RDMAP enables both ULP and LLP security. It requires 2239 that all of its data access be enabled and authorized by the ULP. 2240 It provides no operations for the ULP to gain permissions not 2241 already granted by the host operating system. It allows and 2242 encourages local interactions to specify even more precise 2243 security checks on STag binding and data transfer operations. 2245 By remaining independent of ULP and LLP security protocols, RDMAP 2246 will benefit from continuing improvements at those layers. Users 2247 are provided flexibility to adapt to their specific security 2248 requirements and the ability to adapt to future security 2249 challenges. 2251 11 References 2253 11.1 Normative References 2255 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2256 Requirement Levels", BCP 14, RFC 2119, March 1997. 2258 [VERBS] J. Hilland, ��RDMA Protocol Verbs Specification��, draft- 2259 hilland-rddp-verbs-00. 2261 [DDP] H. Shah et al., "Direct Data Placement over Reliable 2262 Transports", draft-ietf-rddp-ddp-03.txt, February 2005. 2264 [MPA] P. Culley et al., "Marker PDU Aligned Framing for TCP 2265 Specification", draft-ietf-rddp-mpa-01.txt, January 2005. 2267 [SCTP] R. Stewart et al., "Stream Control Transmission Protocol", 2268 RFC 2960, October 2000. 2270 [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, 2271 September 1981. 2273 [RDMASEC] J. Pinkerton et al., "DDP/RDMAP Security", draft-ietf- 2274 rddp-security-05.txt, March 2005. 2276 11.2 Informative References 2278 [RFC2401] Atkinson, R., Kent, S., "Security Architecture for the 2279 Internet Protocol", RFC 2401, November 1998. 2281 [TLS] Dierks, T. and C. Allen, "The TLS Protocol Version 1.0", 2282 RFC 2246, November 1998. 2284 12 Appendix 2286 12.1 DDP Segment Formats for RDMA Messages 2288 This appendix is for information only and is NOT part of the 2289 standard. It simply depicts the DDP Segment format for the various 2290 RDMA Messages. 2292 12.1.1 DDP Segment for RDMA Write 2294 The following figure depicts an RDMA Write, DDP Segment: 2296 0 1 2 3 2297 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2298 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2299 | DDP Control | RDMA Control | 2300 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2301 | Data Sink STag | 2302 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2303 | Data Sink Tagged Offset | 2304 + + 2305 | | 2306 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2307 | RDMA Write ULP Payload | 2308 // // 2309 | | 2310 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2311 Figure 11 RDMA Write, DDP Segment format 2313 12.1.2 DDP Segment for RDMA Read Request 2315 The following figure depicts an RDMA Read Request, DDP Segment: 2317 0 1 2 3 2318 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2319 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2320 | DDP Control | RDMA Control | 2321 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2322 | Reserved (Not Used) | 2323 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2324 | DDP (RDMA Read Request) Queue Number | 2325 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2326 | DDP (RDMA Read Request) Message Sequence Number | 2327 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2328 | DDP (RDMA Read Request) Message Offset | 2329 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2330 | Data Sink STag (SinkSTag) | 2331 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2332 | | 2333 + Data Sink Tagged Offset (SinkTO) + 2334 | | 2335 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2336 | RDMA Read Message Size (RDMARDSZ) | 2337 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2338 | Data Source STag (SrcSTag) | 2339 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2340 | | 2341 + Data Source Tagged Offset (SrcTO) + 2342 | | 2343 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2344 Figure 12 RDMA Read Request, DDP Segment format 2346 12.1.3 DDP Segment for RDMA Read Response 2348 The following figure depicts an RDMA Read Response, DDP Segment: 2350 0 1 2 3 2351 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2352 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2353 | DDP Control | RDMA Control | 2354 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2355 | Data Sink STag | 2356 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2357 | Data Sink Tagged Offset | 2358 + + 2359 | | 2360 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2361 | RDMA Read Response ULP Payload | 2362 // // 2363 | | 2364 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2365 Figure 13 RDMA Read Response, DDP Segment format 2367 12.1.4 DDP Segment for Send and Send with Solicited Event 2369 The following figure depicts a Send and Send with Solicited 2370 Request, DDP Segment: 2372 0 1 2 3 2373 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2374 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2375 | DDP Control | RDMA Control | 2376 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2377 | Reserved (Not Used) | 2378 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2379 | (Send) Queue Number | 2380 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2381 | (Send) Message Sequence Number | 2382 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2383 | (Send) Message Offset | 2384 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2385 | Send ULP Payload | 2386 // // 2387 | | 2388 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2390 Figure 14 Send and Send with Solicited Event, DDP Segment format 2392 12.1.5 DDP Segment for Send with Invalidate and Send with SE and 2393 Invalidate 2395 The following figure depicts a Send with invalidate and Send with 2396 Solicited and Invalidate Request, DDP Segment: 2398 0 1 2 3 2399 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2400 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2401 | DDP Control | RDMA Control | 2402 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2403 | Invalidate STag | 2404 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2405 | (Send) Queue Number | 2406 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2407 | (Send) Message Sequence Number | 2408 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2409 | (Send) Message Offset | 2410 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2411 | Send ULP Payload | 2412 // // 2413 | | 2414 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2415 Figure 15 Send with Invalidate and Send with SE and Invalidate, 2416 DDP Segment 2418 12.1.6 DDP Segment for Terminate 2420 The following figure depicts a Terminate, DDP Segment: 2422 0 1 2 3 2423 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2424 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2425 | DDP Control | RDMA Control | 2426 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2427 | Reserved (Not Used) | 2428 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2429 | DDP (Terminate) Queue Number | 2430 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2431 | DDP (Terminate) Message Sequence Number | 2432 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2433 | DDP (Terminate) Message Offset | 2434 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2435 | Terminate Control | Reserved | 2436 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2437 | DDP Segment Length (if any) | | 2438 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 2439 | | 2440 + + 2441 | Terminated DDP Header (if any) | 2442 + + 2443 | | 2444 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2445 | | 2446 // // 2447 | Terminated RDMA Header (if any) | 2448 + + 2449 | | 2450 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 2451 Figure 16 Terminate, DDP Segment format 2453 12.2 Ordering and Completion Table 2455 The following table summarizes the ordering relationships that are 2456 defined in section 7.5 Ordering and Completions from the 2457 standpoint of the local peer issuing the two Operations. Note, in 2458 the table that follows Send includes Send, Send with Invalidate, 2459 Send with Solicited Event, and Send with Solicited Event and 2460 Invalidate 2462 ------+-------+----------------+----------------+---------------- 2463 First | Later | Placement | Placement | Ordering 2464 Op | Op | guarantee at | guarantee | guarantee at 2465 | | Remote Peer | Local Peer | Remote Peer 2466 | | | | 2467 ------+-------+----------------+----------------+---------------- 2468 Send | Send | No placement | Not applicable | Completed in 2469 | | guarantee. If | | order. 2470 | | guarantee is | | 2471 | | necessary, see | | 2472 | | footnote 1. | | 2473 ------+-------+----------------+----------------+---------------- 2474 Send | RDMA | No placement | Not applicable | Not applicable 2475 | Write | guarantee. If | | 2476 | | guarantee is | | 2477 | | necessary, see | | 2478 | | footnote 1. | | 2479 ------+-------+----------------+----------------+---------------- 2480 Send | RDMA | No placement | RDMA Read | RDMA Read 2481 | Read | guarantee | Response | Response 2482 | | between Send | Payload will | Message will 2483 | | Payload and | not be placed | not be 2484 | | RDMA Read | at the local | generated until 2485 | | Request Header | peer until the | Send has been 2486 | | | Send Payload is| Completed 2487 | | | placed at the | 2488 | | | remote peer | 2489 ------+-------+----------------+----------------+---------------- 2490 RDMA | Send | No placement | Not applicable | Not applicable 2491 Write | | guarantee. If | | 2492 | | guarantee is | | 2493 | | necessary, see | | 2494 | | footnote 1. | | 2495 ------+-------+----------------+----------------+---------------- 2496 RDMA | RDMA | No placement | Not applicable | Not applicable 2497 Write | Write | guarantee. If | | 2498 | | guarantee is | | 2499 | | necessary, see | | 2500 | | footnote 1. | | 2501 ------+-------+----------------+----------------+---------------- 2502 RDMA | RDMA | No placement | RDMA Read | Not applicable 2503 Write | Read | guarantee | Response | 2504 | | between RDMA | Payload will | 2505 | | Write Payload | not be placed | 2506 | | and RDMA Read | at the local | 2507 | | Request Header | peer until the | 2508 | | | RDMA Write | 2509 | | | Payload is | 2510 | | | placed at the | 2511 | | | remote peer | 2512 ------+-------+----------------+----------------+---------------- 2513 RDMA | Send | No placement | Send Payload | Not applicable 2514 Read | | guarantee | may be placed | 2515 | | between RDMA | at the remote | 2516 | | Read Request | peer before the| 2517 | | Header and Send| RDMA Read | 2518 | | payload | Response is | 2519 | | | generated. | 2520 | | | If guarantee is| 2521 | | | necessary, see | 2522 | | | footnote 2. | 2523 ------+-------+----------------+----------------+---------------- 2524 RDMA | RDMA | No placement | RDMA Write | Not applicable 2525 Read | Write | guarantee | Payload may be | 2526 | | between RDMA | placed at the | 2527 | | Read Request | remote peer | 2528 | | Header and RDMA| before the RDMA| 2529 | | Write payload | Read Response | 2530 | | | is generated. | 2531 | | | If guarantee is| 2532 | | | necessary, see | 2533 | | | footnote 2. | 2534 ------+-------+----------------+----------------+---------------- 2535 RDMA | RDMA | No placement | No placement | Second RDMA 2536 Read | Read | guarantee of | guarantee of | Read Response 2537 | | the two RDMA | the two RDMA | will not be 2538 | | Read Request | Read Response | generated until 2539 | | Headers | Payloads. | first RDMA Read 2540 | | Additionally, | | Response is 2541 | | there is no | | generated. 2542 | | guarantee that | | 2543 | | the Tagged | | 2544 | | Buffers | | 2545 | | referenced in | | 2546 | | the RDMA Read | | 2547 | | will be read in| | 2548 | | order | | 2549 Figure 17 Operation Ordering 2551 Footnote 1: If the guarantee is necessary, a ULP may insert an 2552 RDMA Read Operation and wait for it to complete to act as a Fence. 2554 Footnote 2: If the guarantee is necessary, a ULP may wait for the 2555 RDMA Read Operation to complete before performing the Send. 2557 13 Authors Addresses 2559 Paul R. Culley 2560 Hewlett-Packard Company 2561 20555 SH 249 2562 Houston, Tx. USA 77070-2698 2563 Phone: 281-514-5543 2564 Email: paul.culley@hp.com 2566 Dave Garcia 2567 Hewlett-Packard Company 2568 19333 Vallco Parkway 2569 Cupertino, Ca. USA 95014 2570 Phone: 408.285.6116 2571 Email: dave.garcia@hp.com 2573 Jeff Hilland 2574 Hewlett-Packard Company 2575 20555 SH 249 2576 Houston, Tx. USA 77070-2698 2577 Phone: 281-514-9489 2578 Email: jeff.hilland@hp.com 2580 Renato J. Recio 2581 IBM Corp. 2582 11501 Burnett Road 2583 Austin, Tx. USA 78758 2584 Phone: 512-838-3685 2585 Email: recio@us.ibm.com 2586 14 Acknowledgments 2588 Dwight Barron 2589 Hewlett-Packard Company 2590 20555 SH 249 2591 Houston, Tx. USA 77070-2698 2592 Phone: 281-514-2769 2593 Email: dwight.barron@compaq.com 2595 Caitlin Bestler 2596 Email: cait@asomi.com 2598 John Carrier 2599 Adaptec, Inc. 2600 691 S. Milpitas Blvd. 2601 Milpitas, CA 95035 USA 2602 Phone: +1 (360) 378-8526 2603 Email: john_carrier@adaptec.com 2605 Ted Compton 2606 EMC Corporation 2607 Research Triangle Park, NC 27709, USA 2608 Phone: 919-248-6075 2609 Email: compton_ted@emc.com 2611 Uri Elzur 2612 Broadcom Corporation 2613 16215 Alton Parkway 2614 Irvine, California 92619-7013 USA 2615 Phone: +1 (949) 585-6432 2616 Email: Uri@Broadcom.com 2618 Hari Ghadia 2619 Adaptec, Inc. 2620 691 S. Milpitas Blvd., 2621 Milpitas, CA 95035 USA 2622 Phone: +1 (408) 957-5608 2623 Email: hari_ghadia@adaptec.com 2625 Howard C. Herbert 2626 Intel Corporation 2627 MS CH7-404 2628 5000 West Chandler Blvd. 2630 Chandler, Arizona 85226 2631 Phone: 480-554-3116 2632 Email: howard.c.herbert@intel.com 2634 Mike Ko 2635 IBM 2636 650 Harry Rd. 2637 San Jose, CA 95120 2638 Phone: (408) 927-2085 2639 Email: mako@us.ibm.com 2641 Mike Krause 2642 Hewlett-Packard Company 2643 43LN 2644 19410 Homestead Road 2645 Cupertino, CA 95014 USA 2646 Phone: 408-447-3191 2647 Email: krause@cup.hp.com 2649 Dave Minturn 2650 Intel Corporation 2651 MS JF1-210 2652 5200 North East Elam Young Parkway 2653 Hillsboro, Oregon 97124 2654 Phone: 503-712-4106 2655 Email: dave.b.minturn@intel.com 2657 Mike Penna 2658 Broadcom Corporation 2659 16215 Alton Parkway 2660 Irvine, California 92619-7013 USA 2661 Phone: +1 (949) 926-7149 2662 Email: MPenna@Broadcom.com 2664 Jim Pinkerton 2665 Microsoft, Inc. 2666 One Microsoft Way 2667 Redmond, WA, USA 98052 2668 Email: jpink@microsoft.com 2670 Hemal Shah 2671 Intel Corporation 2672 MS PTL1 2673 1501 South Mopac Expressway, #400 2674 Austin, Texas 78746 2675 Phone: 512-732-3963 2676 Email: hemal.shah@intel.com 2678 Allyn Romanow 2679 Cisco Systems 2680 170 W Tasman Drive 2681 San Jose, CA 95134 USA 2682 Phone: +1 408 525 8836 2683 Email: allyn@cisco.com 2685 Tom Talpey 2686 Network Appliance 2687 375 Totten Pond Road 2688 Waltham, MA 02451 USA 2689 Phone: +1 (781) 768-5329 2690 EMail: thomas.talpey@netapp.com 2692 Patricia Thaler 2693 Agilent Technologies, Inc. 2694 1101 Creekside Ridge Drive, #100 2695 M/S-RG10 2696 Roseville, CA 95678 2697 Phone: +1-916-788-5662 2698 email: pat_thaler@agilent.com 2700 Jim Wendt 2701 Hewlett-Packard Company 2702 8000 Foothills Boulevard MS 5668 2703 Roseville, CA 95747-5668 USA 2704 Phone: +1 916 785 5198 2705 Email: jim_wendt@hp.com 2707 15 Full Copyright Statement 2709 This document and the information contained herein is provided on 2710 an ��AS IS�� basis and ADAPTEC INC., AGILENT TECHNOLOGIES INC., 2711 BROADCOM CORPORATION, CISCO SYSTEMS INC., EMC CORPORATION, 2712 HEWLETT-PACKARD COMPANY, INTERNATIONAL BUSINESS MACHINES 2713 CORPORATION, INTEL CORPORATION, MICROSOFT CORPORATION, NETWORK 2714 APPLIANCE INC., THE INTERNET SOCIETY, AND THE INTERNET ENGINEERING 2715 TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 2716 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 2717 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 2718 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 2720 Copyright (c) 2002, 2003, 2004 ADAPTEC INC., BROADCOM CORPORATION, 2721 CISCO SYSTEMS INC., EMC CORPORATION, HEWLETT-PACKARD COMPANY, 2722 INTERNATIONAL BUSINESS MACHINES CORPORATION, INTEL CORPORATION, 2723 MICROSOFT CORPORATION, NETWORK APPLIANCE INC., All Rights 2724 Reserved.