idnits 2.17.1 draft-ietf-rddp-ddp-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 20. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1792. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1762. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1769. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1775. ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There is 1 instance of lines with non-ascii characters in the document. == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 2392 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC 3723' is mentioned on line 1222, but not defined == Unused Reference: 'RFC2119' is defined on line 1509, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2406 (Obsoleted by RFC 4303, RFC 4305) ** Obsolete normative reference: RFC 2407 (Obsoleted by RFC 4306) ** Obsolete normative reference: RFC 2409 (Obsoleted by RFC 4306) == Outdated reference: A later version (-08) exists of draft-ietf-rddp-mpa-04 == Outdated reference: A later version (-07) exists of draft-ietf-rddp-rdmap-06 ** Obsolete normative reference: RFC 2960 (ref. 'SCTP') (Obsoleted by RFC 4960) == Outdated reference: A later version (-07) exists of draft-ietf-rddp-sctp-04 ** Obsolete normative reference: RFC 793 (ref. 'TCP') (Obsoleted by RFC 9293) -- Obsolete informational reference (is this intentional?): RFC 4347 (ref. 'DTLS') (Obsoleted by RFC 6347) -- Obsolete informational reference (is this intentional?): RFC 2401 (ref. 'IPSEC') (Obsoleted by RFC 4301) -- Obsolete informational reference (is this intentional?): RFC 4346 (ref. 'TLS') (Obsoleted by RFC 5246) Summary: 8 errors (**), 0 flaws (~~), 8 warnings (==), 10 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Remote Direct Data Placement Hemal Shah 3 Working Group Broadcom Corporation 4 INTERNET-DRAFT James Pinkerton 5 Category: Standards Track Microsoft Corporation 6 draft-ietf-rddp-ddp-07.txt Renato Recio 7 IBM Corporation 8 Paul Culley 9 Hewlett-Packard Company 11 Expires: March, 2007 September, 2006 13 Direct Data Placement over Reliable Transports 15 Status of this Memo 17 By submitting this Internet-Draft, each author represents that any 18 applicable patent or other IPR claims of which he or she is aware 19 have been or will be disclosed, and any of which he or she becomes 20 aware will be disclosed, in accordance with Section 6 of BCP 79. 22 Internet-Drafts are working documents of the Internet Engineering 23 Task Force (IETF), its areas, and its working groups. Note that 24 other groups may also distribute working documents as Internet- 25 Drafts. 27 Internet-Drafts are draft documents valid for a maximum of six 28 months and may be updated, replaced, or obsoleted by other documents 29 at any time. It is inappropriate to use Internet-Drafts as 30 reference material or to cite them other than as "work in progress." 32 The list of current Internet-Drafts can be accessed at 33 http://www.ietf.org/1id-abstracts.html. 35 The list of Internet-Draft Shadow Directories can be accessed at 36 http://www.ietf.org/shadow.html. 38 Abstract 40 The Direct Data Placement protocol provides information to Place the 41 incoming data directly into an upper layer protocol's receive buffer 42 without intermediate buffers. This removes excess CPU and memory 43 utilization associated with transferring data through the 44 intermediate buffers. 46 Shah, et. al. Expires March 2007 1 47 Table of Contents 49 Status of this Memo ....................................... 1 50 Abstract................................................. 1 51 1 Introduction......................................... 4 52 1.1 Architectural Goals................................... 4 53 1.2 Protocol Overview .................................... 5 54 1.3 DDP Layering......................................... 6 55 2 Glossary............................................ 9 56 2.1 General............................................. 9 57 2.2 LLP.................................................10 58 2.3 Direct Data Placement (DDP)............................11 59 3 Reliable Delivery LLP Requirements......................13 60 4 Header Format........................................15 61 4.1 DDP Control Field ....................................15 62 4.2 DDP Tagged Buffer Model Header.........................16 63 4.3 DDP Untagged Buffer Model Header .......................17 64 4.4 DDP Segment Format....................................18 65 5 Data Transfer........................................19 66 5.1 DDP Tagged or Untagged Buffer Models....................19 67 5.1.1 Tagged Buffer Model.................................19 68 5.1.2 Untagged Buffer Model................................19 69 5.2 Segmentation and Reassembly of a DDP Message.............19 70 5.3 Ordering Among DDP Messages............................21 71 5.4 DDP Message Completion & Delivery.......................22 72 6 DDP Stream Setup & Teardown............................23 73 6.1 DDP Stream Setup.....................................23 74 6.2 DDP Stream Teardown...................................23 75 6.2.1 DDP Graceful Teardown................................23 76 6.2.2 DDP Abortive Teardown................................24 77 7 Error Semantics......................................25 78 7.1 Errors detected at the Data Sink .......................25 79 7.2 DDP Error Numbers ....................................26 80 8 Security Considerations...............................27 81 8.1 Protocol-specific Security Considerations................27 82 8.2 Association of an STag and a DDP Stream.................27 83 8.3 Security Requirements.................................28 84 8.3.1 RNIC Requirements...................................29 85 8.3.2 Privileged Resources Manager Requirement...............30 86 8.4 Security Services for DDP..............................30 87 8.4.1 Available Security Services..........................30 88 8.4.2 Requirements for IPsec Services for DDP................31 89 9 IANA Considerations...................................33 90 10 References...........................................34 91 10.1 Normative References ................................34 92 10.2 Informative References...............................34 93 11 Appendix............................................36 94 11.1 Receive Window sizing................................36 95 12 Authors' Addresses....................................37 96 13 Contributors.........................................38 97 14 Intellectual Property Statement........................41 98 15 Copyright Notice.....................................42 100 Shah, et. al. Expires March 2007 2 101 Table of Figures 103 Figure 1 DDP Layering...................................... 7 104 Figure 2 MPA, DDP, and RDMAP Header Alignment................. 8 105 Figure 3 DDP Control Field.................................15 106 Figure 4 Tagged Buffer DDP Header...........................16 107 Figure 5 Untagged Buffer DDP Header .........................17 108 Figure 6 DDP Segment Format ................................18 110 Shah, et. al. Expires March 2007 3 111 1 Introduction 113 Direct Data Placement Protocol (DDP) enables an Upper Layer Protocol 114 (ULP) to send data to a Data Sink without requiring the Data Sink to 115 Place the data in an intermediate buffer - thus when the data 116 arrives at the Data Sink, the network interface can Place the data 117 directly into the ULP's buffer. This can enable the Data Sink to 118 consume substantially less memory bandwidth than a buffered model 119 because the Data Sink is not required to move the data from the 120 intermediate buffer to the final destination. Additionally, this can 121 also enable the network protocol to consume substantially fewer CPU 122 cycles than if the CPU was used to move the data, and removes the 123 bandwidth limitation of only being able to move data as fast as the 124 CPU can copy the data. 126 DDP preserves ULP record boundaries (messages) while providing a 127 variety of data transfer mechanisms and completion mechanisms to be 128 used to transfer ULP messages. 130 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 131 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 132 document are to be interpreted as described in RFC 2119. 134 1.1 Architectural Goals 136 DDP has been designed with the following high-level architectural 137 goals: 139 * Provide a buffer model that enables the Local Peer to Advertise 140 a named buffer (i.e., a Tag for a buffer) to the Remote Peer, 141 such that across the network the Remote Peer can Place data 142 into the buffer at Remote Peer specified locations. This is 143 referred to as the Tagged Buffer Model. 145 * Provide a second receive buffer model which preserves ULP 146 message boundaries from the Remote Peer and keeps the Local 147 Peer's buffers anonymous (i.e., Untagged). This is referred to 148 as the Untagged Buffer Model. 150 * Provide reliable, in-order Delivery semantics for both Tagged 151 and Untagged Buffer Models. 153 * Provide segmentation and reassembly of ULP messages. 155 * Enable the ULP buffer to be used as a reassembly buffer, 156 without a need for a copy, even if incoming DDP Segments arrive 157 out of order. This requires the protocol to separate Data 158 Placement of ULP Payload contained in an incoming DDP Segment 159 from Data Delivery of completed ULP Messages. 161 * If the Lower Layer Protocol (LLP) supports multiple LLP Streams 162 within a LLP Connection, provide the above capabilities 164 Shah, et. al. Expires March 2007 4 165 independently on each LLP Stream and enable the capability to 166 be exported on a per LLP Stream basis to the ULP. 168 1.2 Protocol Overview 170 DDP supports two basic data transfer models - a Tagged Buffer data 171 transfer model and an Untagged Buffer data transfer model. 173 The Tagged Buffer data transfer model requires the Data Sink to send 174 the Data Source an identifier for the ULP buffer, referred to as a 175 Steering Tag (STag). The STag is transferred to the Data Source 176 using a ULP defined method. Once the Data Source ULP has an STag for 177 a destination ULP buffer, it can request that DDP send the ULP data 178 to the destination ULP buffer by specifying the STag to DDP. Note 179 that the Tagged Buffer does not have to be filled starting at the 180 beginning of the ULP buffer. The ULP Data Source can provide an 181 arbitrary offset into the ULP buffer. 183 The Untagged Buffer data transfer model enables data transfer to 184 occur without requiring the Data Sink to Advertise a ULP Buffer to 185 the Data Source. The Data Sink can queue up a series of receive ULP 186 buffers. An Untagged DDP Message from the Data Source consumes an 187 Untagged Buffer at the Data Sink. Because DDP is message oriented, 188 even if the Data Source sends a DDP Message payload smaller than the 189 receive ULP buffer, the partially filled receive ULP buffer is 190 Delivered to the ULP anyway. If the Data Source sends a DDP Message 191 payload larger than the receive ULP buffer, it results in an error. 193 There are several key differences between the Tagged and Untagged 194 Buffer Model: 196 * For the Tagged Buffer Model, the Data Source specifies which 197 received Tagged Buffer will be used for a specific Tagged DDP 198 Message (sender-based ULP buffer management). For the Untagged 199 Buffer Model, the Data Sink specifies the order in which 200 Untagged Buffers will be consumed as Untagged DDP Messages are 201 received (receiver-based ULP buffer management). 203 * For the Tagged Buffer Model, the ULP at the Data Sink must 204 Advertise the ULP buffer to the Data Source through a ULP 205 specific mechanism before data transfer can occur. For the 206 Untagged Buffer Model, data transfer can occur without an end- 207 to-end explicit ULP buffer Advertisement. Note, however, that 208 the ULP needs to address flow control issues. 210 * For the Tagged Buffer Model, a DDP Message can start at an 211 arbitrary offset within the Tagged Buffer. For the Untagged 212 Buffer Model, a DDP Message can only start at offset 0. 214 * The Tagged Buffer Model allows multiple DDP Messages targeted 215 to a Tagged Buffer with a single ULP buffer Advertisement. The 216 Untagged Buffer Model requires associating a receive ULP buffer 217 for each DDP Message targeted to an Untagged Buffer. 219 Shah, et. al. Expires March 2007 5 220 Either data transfer model Places a ULP Message into a DDP Message. 221 Each DDP Message is then sliced into DDP Segments that are intended 222 to fit within a lower-layer-protocol's (LLP) Maximum Upper Layer 223 Protocol Data Unit (MULPDU). Thus the ULP can post arbitrary size 224 ULP Messages, containing up to 2^32 - 1 octets of ULP Payload, and 225 DDP slices the ULP message into DDP Segments which are reassembled 226 transparently at the Data Sink. 228 DDP provides in-order Delivery for the ULP. However, DDP 229 differentiates between Data Delivery and Data Placement. DDP 230 provides enough information in each DDP Segment to allow the ULP 231 Payload in each inbound DDP Segment payloads to be directly Placed 232 into the correct ULP Buffer, even when the DDP Segments arrive out- 233 of-order. Thus, DDP enables the reassembly of ULP Payload contained 234 in DDP Segments of a DDP Message into a ULP Message to occur within 235 the ULP Buffer, therefore eliminating the traditional copy out of 236 the reassembly buffer into the ULP Buffer. 238 A DDP Message's payload is Delivered to the ULP when: 240 * all DDP Segments of a DDP Message have been completely received 241 and the payload of the DDP Message has been Placed into the 242 associated ULP Buffer, 244 * all prior DDP Messages have been Placed, and 246 * all prior DDP Message Deliveries have been performed. 248 The LLP under DDP may support a single LLP Stream of data per 249 connection (e.g., TCP [TCP]) or multiple LLP Streams of data per 250 connection (e.g., SCTP [SCTP]). But in either case, DDP is specified 251 such that each DDP Stream is independent and maps to a single LLP 252 Stream. Within a specific DDP Stream, the LLP Stream is required to 253 provide in-order, reliable Delivery. Note that DDP has no ordering 254 guarantees between DDP Streams. 256 A DDP protocol could potentially run over reliable Delivery LLPs or 257 unreliable Delivery LLPs. This specification requires reliable, in 258 order Delivery LLPs. 260 1.3 DDP Layering 262 DDP is intended to be LLP independent, subject to the requirements 263 defined in section 3. However, DDP was specifically defined to be 264 part of a family of protocols that were created to work well 265 together, as shown in Figure 1 DDP Layering. For LLP protocol 266 definitions of each LLP, see Marker PDU Aligned Framing for TCP 267 Specification [MPA] and Stream Control Transmission Protocol (SCTP) 268 Direct Data Placement (DDP) Adaptation [SCTPDDP]. 270 DDP enables direct data Placement capability for any ULP, but it has 271 been specifically designed to work well with Remote Direct Memory 273 Shah, et. al. Expires March 2007 6 274 Access Protocol (RDMAP) (see [RDMAP]), and is part of the iWARP 275 protocol suite. 277 +-------------------+ 278 | | 279 | RDMA ULP | 280 | | 281 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 282 | | | 283 | ULP | RDMAP | 284 | | | 285 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 286 | | 287 | DDP protocol | 288 | | 289 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 290 | | | 291 | MPA | | 292 | | | 293 | | | 294 +-+-+-+-+-+-+-+-+-+ SCTP | 295 | | | 296 | TCP | | 297 | | | 298 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 300 Figure 1 DDP Layering 302 If DDP is layered below RDMAP and on top of MPA and TCP, then the 303 respective headers and payload are arranged as follows (Note: For 304 clarity, MPA header and CRC are included but framing markers are not 305 shown.): 307 Shah, et. al. Expires March 2007 7 308 0 1 2 3 309 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 310 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 311 | | 312 // TCP Header // 313 | | 314 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 315 | MPA Header | | 316 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + 317 | | 318 // DDP Header // 319 | | 320 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 321 | | 322 // RDMAP Header // 323 | | 324 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 325 | | 326 // // 327 // RDMAP ULP Payload // 328 // // 329 | | 330 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 331 | MPA CRC | 332 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 334 Figure 2 MPA, DDP, and RDMAP Header Alignment 336 Shah, et. al. Expires March 2007 8 337 2 Glossary 339 2.1 General 341 Advertisement (Advertised, Advertise, Advertisements, Advertises) - 342 The act of informing a Remote Peer that a local RDMA Buffer is 343 available to it. A Node makes available an RDMA Buffer for 344 incoming RDMA Read or RDMA Write access by informing its 345 RDMA/DDP peer of the Tagged Buffer identifiers (STag, base 346 address, length). This advertisement of Tagged Buffer 347 information is not defined by RDMA/DDP and is left to the ULP. A 348 typical method would be for the Local Peer to embed the Tagged 349 Buffer's Steering Tag, address, and length in a Send message 350 destined for the Remote Peer. 352 Data Delivery (Delivery, Delivered, Delivers) - Delivery is defined 353 as the process of informing the ULP or consumer that a 354 particular message is available for use. This is specifically 355 different from "Placement", which may generally occur in any 356 order, while the order of "Delivery" is strictly defined. See 357 "Data Placement". 359 Data Sink - The peer receiving a data payload. Note that the Data 360 Sink can be required to both send and receive RDMA/DDP Messages 361 to transfer a data payload. 363 Data Source - The peer sending a data payload. Note that the Data 364 Source can be required to both send and receive RDMA/DDP 365 Messages to transfer a data payload. 367 Delivery - See Data Delivery in Section 2.1. 369 Delivered - See Data Delivery in Section 2.1. 371 Delivers - See Data Delivery in Section 2.1. 373 iWARP - A suite of wire protocols comprised of RDMAP [RDMAP], DDP 374 (this specification), and Marker PDU Aligned Framing for TCP 375 (MPA) [MPA]. The iWARP protocol suite may be layered above TCP, 376 SCTP, or other transport protocols. 378 Local Peer - The RDMA/DDP protocol implementation on the local end 379 of the connection. Used to refer to the local entity when 380 describing a protocol exchange or other interaction between two 381 Nodes. 383 Node - A computing device attached to one or more links of a 384 network. A Node in this context does not refer to a specific 385 application or protocol instantiation running on the computer. A 386 Node may consist of one or more RDMA Enabled Network Interface 387 Controllers (RNICs) installed in a host computer. 389 Placement - See "Data Placement" in Section 2.3 391 Shah, et. al. Expires March 2007 9 392 Placed - See "Data Placement" in Section 2.3 394 Places - See "Data Placement" in Section 2.3 396 Remote Peer - The RDMA/DDP protocol implementation on the opposite 397 end of the connection. Used to refer to the remote entity when 398 describing protocol exchanges or other interactions between two 399 Nodes. 401 RNIC - RDMA Enabled Network Interface Controller. In this context, 402 this would be a network I/O adapter or embedded controller with 403 iWARP functionality. 405 ULP - Upper Layer Protocol. The protocol layer above the protocol 406 layer currently being referenced. The ULP for RDMA/DDP is 407 expected to be an Operating System (OS), application, adaptation 408 layer, or proprietary device. The RDMA/DDP documents do not 409 specify a ULP - they provide a set of semantics that allow a ULP 410 to be designed to utilize RDMA/DDP. 412 ULP Message - The ULP data that is handed to a specific protocol 413 layer for transmission. Data boundaries are preserved as they 414 are transmitted through iWARP. 416 ULP Payload - The ULP data that is contained within a single 417 protocol segment or packet (e.g., a DDP Segment). 419 2.2 LLP 421 LLP - Lower Layer Protocol. The protocol layer beneath the protocol 422 layer currently being referenced. For example, for DDP the LLP 423 is SCTP DDP Adaptation, MPA, or other transport protocols. For 424 RDMA, the LLP is DDP. 426 LLP Connection - Corresponds to an LLP transport-level connection 427 between the peer LLP layers on two nodes. 429 LLP Stream - Corresponds to a single LLP transport-level stream 430 between the peer LLP layers on two Nodes. One or more LLP 431 Streams may map to a single transport-level LLP Connection. For 432 transport protocols that support multiple streams per connection 433 (e.g., SCTP), a LLP Stream corresponds to one transport-level 434 stream. 436 MULPDU - Maximum Upper Layer Protocol Data Unit (ULPDU). The current 437 maximum size of the record that is acceptable for DDP to pass to 438 the LLP for transmission. 440 ULPDU - Upper Layer Protocol Data Unit. The data record defined by 441 the layer above MPA. 443 Shah, et. al. Expires March 2007 10 444 2.3 Direct Data Placement (DDP) 446 Data Placement (Placement, Placed, Places) - For DDP, this term is 447 specifically used to indicate the process of writing to a data 448 buffer by a DDP implementation. DDP Segments carry Placement 449 information, which may be used by the receiving DDP 450 implementation to perform Data Placement of the DDP Segment ULP 451 Payload. See "Data Delivery" and "Direct Data Placement". 453 DDP Abortive Teardown - The act of closing a DDP Stream without 454 attempting to complete in-progress and pending DDP Messages. 456 DDP Graceful Teardown - The act of closing a DDP Stream such that 457 all in-progress and pending DDP Messages are allowed to complete 458 successfully. 460 DDP Control Field - A fixed 8-bit field in the DDP Header. 462 DDP Header - The header present in all DDP Segments. The DDP Header 463 contains control and Placement fields that are used to define 464 the final Placement location for the ULP Payload carried in a 465 DDP Segment. 467 DDP Message - A ULP defined unit of data interchange, which is 468 subdivided into one or more DDP Segments. This segmentation may 469 occur for a variety of reasons, including segmentation to 470 respect the maximum segment size of the underlying transport 471 protocol. 473 DDP Segment - The smallest unit of data transfer for the DDP 474 protocol. It includes a DDP Header and ULP Payload (if present). 475 A DDP Segment should be sized to fit within the Lower Layer 476 Protocol MULPDU. 478 DDP Stream - a sequence of DDP messages whose ordering is defined by 479 the LLP. For SCTP, a DDP Stream maps directly to an SCTP stream. 480 For MPA, a DDP Stream maps directly to a TCP connection and a 481 single DDP Stream is supported. Note that DDP has no ordering 482 guarantees between DDP Streams. 484 DDP Stream Identifier (ID) - An identifier for a DDP Stream. 486 Direct Data Placement - A mechanism whereby ULP data contained 487 within DDP Segments may be Placed directly into its final 488 destination in memory without processing of the ULP. This may 489 occur even when the DDP Segments arrive out of order. Out of 490 order Placement support may require the Data Sink to implement 491 the LLP and DDP as one functional block. 493 Direct Data Placement Protocol (DDP) - Also, a wire protocol that 494 supports Direct Data Placement by associating explicit memory 495 buffer placement information with the LLP payload units. 497 Shah, et. al. Expires March 2007 11 498 Message Offset (MO) - For the DDP Untagged Buffer Model, specifies 499 the offset, in octets, from the start of a DDP Message. 501 Message Sequence Number (MSN) - For the DDP Untagged Buffer Model, 502 specifies a sequence number that is increasing with each DDP 503 Message. 505 Protection Domain (PD) - A Mechanism used to associate a DDP Stream 506 and an STag. Under this mechanism, the use of an STag is valid 507 on a DDP Stream if the STag has the same Protection Domain 508 Identifier (PD ID) as the DDP Stream. 510 Protection Domain Identifier (PD ID) - An identifier for the 511 Protection Domain. 513 Queue Number (QN) - For the DDP Untagged Buffer Model, identifies a 514 destination Data Sink queue for a DDP Segment. 516 Steering Tag - An identifier of a Tagged Buffer on a Node, valid as 517 defined within a protocol specification. 519 STag - Steering Tag 521 Tagged Buffer - A buffer that is explicitly Advertised to the Remote 522 Peer through exchange of an STag, Tagged Offset, and length. 524 Tagged Buffer Model - A DDP data transfer model used to transfer 525 Tagged Buffers from the Local Peer to the Remote Peer. 527 Tagged DDP Message - A DDP Message that targets a Tagged Buffer. 529 Tagged Offset (TO) - The offset within a Tagged Buffer on a Node. 531 ULP Buffer - A buffer owned above the DDP Layer and advertised to 532 the DDP Layer either as a Tagged Buffer or an Untagged ULP 533 Buffer. 535 ULP Message Length - The total length, in octets, of the ULP Payload 536 contained in a DDP Message. 538 Untagged Buffer - A buffer that is not explicitly Advertised to the 539 Remote Peer. 541 Untagged Buffer Model - A DDP data transfer model used to transfer 542 Untagged Buffers from the Local Peer to the Remote Peer. 544 Untagged DDP Message - A DDP Message that targets an Untagged 545 Buffer. 547 Shah, et. al. Expires March 2007 12 548 3 Reliable Delivery LLP Requirements 550 Any protocol that can serve as an LLP to DDP MUST meet the following 551 requirements. 553 1. LLPs MUST expose MULPDU & MULPDU Changes. This is required so 554 that the DDP layer can perform segmentation aligned with the 555 MULPDU and can adapt as MULPDU changes come about. The corner 556 case of how to handle outstanding requests during a MULPDU 557 change is covered by the requirements below. 559 2. In the event of a MULPDU change, DDP MUST NOT be required by the 560 LLP to re-segment DDP Segments that have been previously posted 561 to the LLP. Note that under pathological conditions the LLP may 562 change the advertised MULPDU more frequently than the queue of 563 previously posted DDP Segment transmit requests is flushed. 564 Under this pathological condition, the LLP transmit queue can 565 contain DDP Messages which were posted multiple MULPDU updates 566 previously, thus there may be no correlation between the queued 567 DDP Segment(s) and the LLP's current value of MULPDU. 569 3. The LLP MUST ensure that if it accepts a DDP Segment, it will 570 transfer it reliably to the receiver or return with an error 571 stating that the transfer failed to complete. 573 4. The LLP MUST preserve DDP Segment and Message boundaries at the 574 Data Sink. 576 5. The LLP MAY provide the incoming segments out of order for 577 Placement, but if it does, it MUST also provide information that 578 specifies what the sender specified order was. 580 6. LLP MUST provide a strong digest (at least equivalent to CRC32- 581 C) to cover at least the DDP Segment. It is believed that some 582 of the existing data integrity digests are not sufficient and 583 that direct memory transfer semantics requires a stronger digest 584 than, for example, a simple checksum. 586 7. On receive, the LLP MUST provide the length of the DDP Segment 587 received. This ensures that DDP does not have to carry a length 588 field in its header. 590 8. If an LLP does not support teardown of a LLP Stream independent 591 of other LLP Streams and a DDP error occurs on a specific DDP 592 Stream, then the LLP MUST label the associated LLP Stream as an 593 erroneous LLP Stream and MUST NOT allow any further data 594 transfer on that LLP Stream after DDP requests the associated 595 DDP Stream to be torn down. 597 9. For a specific LLP Stream, the LLP MUST provide a mechanism to 598 indicate that the LLP Stream has been gracefully torn down. For 599 a specific LLP Connection, the LLP MUST provide a mechanism to 600 indicate that the LLP Connection has been gracefully torn down. 602 Shah, et. al. Expires March 2007 13 603 Note that if the LLP does not allow an LLP Stream to be torn 604 down independently of the LLP Connection, the above requirements 605 allow the LLP to notify DDP of both events at the same time. 607 10. For a specific LLP Connection, when all LLP Streams are either 608 gracefully torn down or are labeled as erroneous LLP Streams, 609 the LLP Connection MUST be torn down. 611 11. The LLP MUST NOT pass a duplicate DDP Segment to the DDP Layer 612 after it has passed all the previous DDP Segments to the DDP 613 Layer and the associated ordering information for the previous 614 DDP Segments and the current DDP Segment. 616 Shah, et. al. Expires March 2007 14 617 4 Header Format 619 DDP has two different header formats: one for Data Placement into 620 Tagged Buffers, and the other for Data Placement into Untagged 621 Buffers. See Section 5.1 for a description of the two models. 623 4.1 DDP Control Field 625 The first 8 bits of the DDP Header carry a DDP Control Field that is 626 common between the two formats. It is shown below in Figure 3, 627 offset by 16 bits to accommodate the MPA header defined in [MPA]. 628 The MPA header is only present if DDP is layered on top of MPA. 630 0 1 2 3 631 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 632 +-+-+-+-+-+-+-+-+ 633 |T|L| Rsvd |DV | 634 +-+-+-+-+-+-+-+-+ 635 Figure 3 DDP Control Field 637 T - Tagged flag: 1 bit. 639 Specifies the Tagged or Untagged Buffer Model. If set to one, 640 the ULP Payload carried in this DDP Segment MUST be Placed into 641 a Tagged Buffer. 643 If set to zero, the ULP Payload carried in this DDP Segment 644 MUST be Placed into an Untagged Buffer. 646 L - Last flag: 1 bit. 648 Specifies whether the DDP Segment is the Last segment of a DDP 649 Message. It MUST be set to one on the last DDP Segment of every 650 DDP Message. It MUST NOT be set to one on any other DDP 651 Segment. 653 The DDP Segment with the L bit set to 1 MUST be posted to the 654 LLP after all other DDP Segments of the associated DDP Message 655 have been posted to the LLP. For an Untagged DDP Message, the 656 DDP Segment with the L bit set to 1 MUST carry the highest MO. 658 If the Last flag is set to one, the DDP Message payload MUST be 659 Delivered to the ULP after: 661 . Placement of all DDP Segments of this DDP Message and all 662 prior DDP Messages, and 664 . Delivery of each prior DDP Message. 666 If the Last flag is set to zero, the DDP Segment is an 667 intermediate DDP Segment. 669 Shah, et. al. Expires March 2007 15 670 Rsvd - Reserved: 4 bits. 672 Reserved for future use by the DDP protocol. This field MUST be 673 set to zero on transmit, and not checked on receive. 675 DV - Direct Data Placement Protocol Version: 2 bits. 677 The version of the DDP Protocol in use. This field MUST be set 678 to one to indicate the version of the specification described 679 in this document. The value of DV MUST be the same for all the 680 DDP Segments transmitted or received on a DDP Stream. 682 4.2 DDP Tagged Buffer Model Header 684 Figure 4 shows the DDP Header format that MUST be used in all DDP 685 Segments that target Tagged Buffers. It includes the DDP Control 686 Field previously defined in Section 4.1. (Note: In Figure 4, the DDP 687 Header is offset by 16 bits to accommodate the MPA header defined in 688 [MPA]. The MPA header is only present if DDP is layered on top of 689 MPA.) 691 0 1 2 3 692 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 693 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 694 |T|L| Rsvd | DV| RsvdULP | 695 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 696 | STag | 697 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 698 | | 699 + TO + 700 | | 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 702 Figure 4 Tagged Buffer DDP Header 704 T is set to one. 706 RsvdULP - Reserved for use by the ULP: 8 bits. 708 The RsvdULP field is opaque to the DDP protocol and can be 709 structured in any way by the ULP. At the Data Source, DDP MUST 710 set RsvdULP Field to the value specified by the ULP. It is 711 transferred unmodified from the Data Source to the Data Sink. 712 At the Data Sink, DDP MUST provide the RsvdULP field to the ULP 713 when the DDP Message is delivered. Each DDP Segment within a 714 specific DDP Message MUST contain the same value for this 715 field. The Data Source MUST ensure that each DDP Segment within 716 a specific DDP Message contains the same value for this field. 718 STag - Steering Tag: 32 bits. 720 The Steering Tag identifies the Data Sink's Tagged Buffer. The 721 STag MUST be valid for this DDP Stream. The STag is associated 722 with the DDP Stream through a mechanism that is outside the 724 Shah, et. al. Expires March 2007 16 725 scope of the DDP Protocol specification. At the Data Source, 726 DDP MUST set the STag field to the value specified by the ULP. 727 At the Data Sink, the DDP MUST provide the STag field when the 728 ULP Message is delivered. Each DDP Segment within a specific 729 DDP Message MUST contain the same value for this field and MUST 730 be the value supplied by the ULP. The Data Source MUST ensure 731 that each DDP Segment within a specific DDP Message contains 732 the same value for this field. 734 TO - Tagged Offset: 64 bits. 736 The Tagged Offset specifies the offset, in octets, within the 737 Data Sink's Tagged Buffer, where the Placement of ULP Payload 738 contained in the DDP Segment starts. A DDP Message MAY start at 739 an arbitrary TO within a Tagged Buffer. 741 4.3 DDP Untagged Buffer Model Header 743 Figure 5 shows the DDP Header format that MUST be used in all DDP 744 Segments that target Untagged Buffers. It includes the DDP Control 745 Field previously defined in Section 4.1. (Note: In Figure 5, the DDP 746 Header is offset by 16 bits to accommodate the MPA header defined in 747 [MPA]. The MPA header is only present if DDP is layered on top of 748 MPA.) 750 0 1 2 3 751 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 752 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 753 |T|L| Rsvd | DV| RsvdULP[0:7] | 754 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 755 | RsvdULP[8:39] | 756 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 757 | QN | 758 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 759 | MSN | 760 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 761 | MO | 762 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 763 Figure 5 Untagged Buffer DDP Header 765 T is set to zero. 767 RsvdULP - Reserved for use by the ULP: 40 bits. 769 The RsvdULP field is opaque to the DDP protocol and can be 770 structured in any way by the ULP. At the Data Source, DDP MUST 771 set RsvdULP Field to the value specified by the ULP. It is 772 transferred unmodified from the Data Source to the Data Sink. 773 At the Data Sink, DDP MUST provide RsvdULP field to the ULP 774 when the ULP Message is Delivered. Each DDP Segment within a 775 specific DDP Message MUST contain the same value for the 777 Shah, et. al. Expires March 2007 17 778 RsvdULP field. At the Data Sink, the DDP implementation is NOT 779 REQUIRED to verify that the same value is present in the 780 RsvdULP field of each DDP Segment within a specific DDP Message 781 and MAY provide the value from any one of the received DDP 782 Segment to the ULP when the ULP Message is Delivered. 784 QN - Queue Number: 32 bits. 786 The Queue Number identifies the Data Sink's Untagged Buffer 787 queue referenced by this header. Each DDP segment within a 788 specific DDP message MUST contain the same value for this field 789 and MUST be the value supplied by the ULP at the Data Source. 790 The Data Source MUST ensure that each DDP Segment within a 791 specific DDP Message contains the same value for this field. 793 MSN - Message Sequence Number: 32 bits. 795 The Message Sequence Number specifies a sequence number that 796 MUST be increased by one (modulo 2^32) with each DDP Message 797 targeting the specific Queue Number on the DDP Stream 798 associated with this DDP Segment. The initial value for MSN 799 MUST be one. The MSN value MUST wrap to 0 after a value of 800 0xFFFFFFFF. Each DDP segment within a specific DDP message MUST 801 contain the same value for this field. The Data Source MUST 802 ensure that each DDP Segment within a specific DDP Message 803 contains the same value for this field. 805 MO - Message Offset: 32 bits. 807 The Message Offset specifies the offset, in octets, from the 808 start of the DDP Message represented by the MSN and Queue 809 Number on the DDP Stream associated with this DDP Segment. The 810 MO referencing the first octet of the DDP Message MUST be set 811 to zero by the DDP layer. 813 4.4 DDP Segment Format 815 Each DDP Segment MUST contain a DDP Header. Each DDP Segment may 816 also contain ULP Payload. Following is the DDP Segment format: 818 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 819 | DDP | | 820 | Header| ULP Payload (if any) | 821 | | | 822 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 823 Figure 6 DDP Segment Format 825 Shah, et. al. Expires March 2007 18 826 5 Data Transfer 828 DDP supports multi-segment DDP Messages. Each DDP Message is 829 composed of one or more DDP Segments. Each DDP Segment contains a 830 DDP Header. The DDP Header contains the information required by the 831 receiver to Place any ULP Payload included in the DDP Segment. 833 5.1 DDP Tagged or Untagged Buffer Models 835 DDP uses two basic Buffer Models for the Placement of the ULP 836 Payload: Tagged Buffer Model and Untagged Buffer Model. 838 5.1.1 Tagged Buffer Model 840 The Tagged Buffer Model is used by the Data Source to transfer a DDP 841 Message into a Tagged Buffer at the Data Sink that has been 842 previously Advertised to the Data Source. An STag identifies a 843 Tagged Buffer. For the Placement of a DDP Message using the Tagged 844 Buffer model, the STag is used to identify the buffer, and the TO is 845 used to identify the offset within the Tagged Buffer into which the 846 ULP Payload is transferred. The protocol used to Advertise the 847 Tagged Buffer is outside the scope of this specification (i.e., ULP 848 specific). A DDP Message can start at an arbitrary TO within a 849 Tagged Buffer. 851 Additionally, a Tagged Buffer can potentially be written multiple 852 times. This might be done for error recovery or because a buffer is 853 being re-used after some ULP specific synchronization mechanism. 855 5.1.2 Untagged Buffer Model 857 The Untagged Buffer Model is used by the Data Source to transfer a 858 DDP Message to the Data Sink into a queued buffer. 860 The DDP Queue Number is used by the ULP to separate ULP messages 861 into different queues of receive buffers. For example, if two queues 862 were supported, the ULP could use one queue to post buffers handed 863 to it by the application above the ULP, and it could use the other 864 queue for buffers which are only consumed by ULP specific control 865 messages. This enables the separation of ULP control messages from 866 opaque ULP Payload when using Untagged Buffers. 868 The DDP Message Sequence Number can be used by the Data Sink to 869 identify the specific Untagged Buffer. The protocol used to 870 communicate how many buffers have been queued is outside the scope 871 of this specification. Similarly, the exact implementation of the 872 buffer queue is outside the scope of this specification. 874 5.2 Segmentation and Reassembly of a DDP Message 876 At the Data Source, the DDP layer MUST segment the data contained in 877 a ULP message into a series of DDP Segments, where each DDP Segment 878 contains a DDP Header and ULP Payload, and MUST be no larger than 880 Shah, et. al. Expires March 2007 19 881 the MULPDU value advertised by the LLP. The ULP Message Length MUST 882 be less than 2^32. At the Data Source, the DDP layer MUST send all 883 the data contained in the ULP message. At the Data Sink, the DDP 884 layer MUST Place the ULP Payload contained in all valid incoming DDP 885 Segments associated with a DDP Message into the ULP Buffer. 887 DDP Message segmentation at the Data Source is accomplished by 888 identifying a DDP Message (which corresponds one-to-one with a ULP 889 Message) uniquely and then, for each associated DDP Segment of a DDP 890 Message, by specifying an octet offset for the portion of the ULP 891 Message contained in the DDP Segment. 893 For an Untagged DDP Message, the combination of the QN and MSN 894 uniquely identifies a DDP Message. The octet offset for each DDP 895 Segment of a Untagged DDP Message is the MO field. For each DDP 896 Segment of a Untagged DDP Message, the MO MUST be set to the octet 897 offset from the first octet in the associated ULP Message (which is 898 defined to be zero) to the first octet in the ULP Payload contained 899 in the DDP Segment. 901 For example, if the ULP Untagged Message was 2048 octets, and the 902 MULPDU was 1500 octets, the Data Source would generate two DDP 903 Segments, one with MO = 0, containing 1482 octets of ULP Payload, 904 and a second with MO = 1482, containing 566 octets of ULP Payload. 905 In this example, the amount of ULP Payload for the first DDP Segment 906 was calculated as: 908 1482 = 1500 (MULPDU) - 18 (for the DDP Header) 910 For a Tagged DDP Message, the STag and TO, combined with the in- 911 order delivery characteristics of the LLP, are used to segment and 912 reassemble the ULP Message. Because the initial octet offset (the TO 913 field) can be non-zero, recovery of the original ULP Message 914 boundary cannot be done in the general case without an additional 915 ULP Message. 917 Implementers Note: One implementation, valid for some ULPs such 918 as RDMAP, is to not directly support recovery of the ULP 919 Message boundary for a Tagged DDP Message. For example, the ULP 920 may wish to have the Local Peer use small buffers at the Data 921 Source even when the ULP at the Data Sink has advertised a 922 single large Tagged Buffer for this data transfer. In this 923 case, the ULP may choose to use the same STag for multiple 924 consecutive ULP Messages. Thus a non-zero initial TO and re-use 925 of the STag effectively enables the ULP to implement 926 segmentation and reassembly due to ULP specific constraints. 927 See [RDMAP] for details of how this is done. 929 A different implementation of a ULP could use an Untagged DDP 930 Message sent after the Tagged DDP Message which details the 931 initial TO for the STag that was used in the Tagged DDP 932 Message. And finally, another implementation of a ULP could 933 choose to always use an initial TO of zero such that no 935 Shah, et. al. Expires March 2007 20 936 additional message is required to convey the initial TO used in 937 a Tagged DDP Message. 939 Regardless of whether the ULP chooses to recover the original ULP 940 Message boundary at the Data Sink for a Tagged DDP Message, DDP 941 supports segmentation and reassembly of the Tagged DDP Message. The 942 STag is used to identify the ULP Buffer at the Data Sink and the TO 943 is used to identify the octet-offset within the ULP Buffer 944 referenced by the STag. The ULP at the Data Source MUST specify the 945 STag and the initial TO when the ULP Message is handed to DDP. 947 For each DDP Segment of a Tagged DDP Message, the TO MUST be set to 948 the octet offset from the first octet in the associated ULP Message 949 to the first octet in the ULP Payload contained in the DDP Segment, 950 plus the TO assigned to the first octet in the associated ULP 951 Message. 953 For example, if the ULP Tagged Message was 2048 octets with an 954 initial TO of 16384, and the MULPDU was 1500 octets, the Data Source 955 would generate two DDP Segments, one with TO = 16384, containing the 956 first 1486 octets of ULP payload, and a second with TO = 17870, 957 containing 562 octets of ULP payload. In this example, the amount of 958 ULP payload for the first DDP Segment was calculated as: 960 1486 = 1500 (MULPDU) - 14 (for the DDP Header) 962 A zero-length DDP Message is allowed and MUST consume exactly one 963 DDP Segment. Only the DDP Control and RsvdULP Fields MUST be valid 964 for a zero length Tagged DDP Segment. The STag and TO fields MUST 965 NOT be checked for a zero-length Tagged DDP Message. 967 For either Untagged or Tagged DDP Messages, the Data Sink is not 968 required to verify that the entire ULP Message has been received. 970 5.3 Ordering Among DDP Messages 972 Messages passed through the DDP MUST conform to the ordering rules 973 defined in this section. 975 At the Data Source, DDP: 977 * MUST transmit DDP Messages in the order they were submitted to 978 the DDP layer, 980 * SHOULD transmit DDP Segments within a DDP Message in increasing 981 MO order for Untagged DDP Messages and in increasing TO order 982 for Tagged DDP Messages. 984 At the Data Sink, DDP (Note: The following rules are motivated by 985 LLP implementations that separate Placement and Delivery.): 987 * MAY perform Placement of DDP Segments out of order, 989 Shah, et. al. Expires March 2007 21 990 * MAY perform Placement of a DDP Segment more than once, 992 * MUST Deliver a DDP Message to the ULP at most once, 994 * MUST Deliver DDP Messages to the ULP in the order they were 995 sent by the Data Source. 997 5.4 DDP Message Completion & Delivery 999 At the Data Source, DDP Message transfer is considered completed 1000 when the reliable, in-order transport LLP has indicated that the 1001 transfer will occur reliably. Note that this in no way restricts the 1002 LLP from buffering the data at either the Data Source or Data Sink. 1003 Thus at the Data Source, completion of a DDP Message does not 1004 necessarily mean that the Data Sink has received the message. 1006 At the Data Sink, DDP MUST Deliver a DDP Message if and only if all 1007 of the following are true: 1009 * the last DDP Segment of the DDP Message had its Last flag set, 1011 * all of the DDP Segments of the DDP Message have been Placed, 1013 * all preceding DDP Messages have been Placed, and 1015 * each preceding DDP Message has been Delivered to the ULP. 1017 At the Data Sink, DDP MUST provide the ULP Message Length to the ULP 1018 when an Untagged DDP Message is Delivered. The ULP Message Length 1019 may be calculated by adding the MO and the ULP Payload length in the 1020 last DDP Segment (with the Last flag set) of an Untagged DDP 1021 Message. 1023 At the Data Sink, DDP MUST provide the RsvdULP Field of the DDP 1024 Message to the ULP when the DDP Message is delivered. 1026 Shah, et. al. Expires March 2007 22 1027 6 DDP Stream Setup & Teardown 1029 This section describes LLP independent issues related to DDP Stream 1030 setup and teardown. 1032 6.1 DDP Stream Setup 1034 It is expected that the ULP will use a mechanism outside the scope 1035 of this specification to establish an LLP Connection, and that the 1036 LLP Connection will support one or more LLP Streams (e.g., MPA/TCP 1037 or SCTP). After the LLP sets up the LLP Stream, it will enable a DDP 1038 Stream on a specific LLP Stream at an appropriate point. 1040 The ULP is required to enable both endpoints of an LLP Stream for 1041 DDP data transfer at the same time, in both directions; this is 1042 necessary so that the Data Sink can properly recognize the DDP 1043 Segments. 1045 6.2 DDP Stream Teardown 1047 DDP MUST NOT independently initiate Stream Teardown. DDP either 1048 responds to a stream being torn down by the LLP or processes a 1049 request from the ULP to teardown a stream. DDP Stream teardown 1050 disables DDP capabilities on both endpoints. For connection-oriented 1051 LLPs, DDP Stream teardown MAY result in underlying LLP Connection 1052 teardown. 1054 6.2.1 DDP Graceful Teardown 1056 It is up to the ULP to ensure that DDP teardown happens on both 1057 endpoints of the DDP Stream at the same time; this is necessary so 1058 that the Data Sink stops trying to interpret the DDP Segments. 1060 If the Local Peer ULP indicates graceful teardown, the DDP layer on 1061 the Local Peer SHOULD ensure that all ULP data would be transferred 1062 before the underlying LLP Stream & Connection are torn down, and any 1063 further data transfer requests by the Local Peer ULP MUST return an 1064 error. 1066 If the DDP layer on the Local Peer receives a graceful teardown 1067 request from the LLP, any further data received after the request is 1068 considered an error and MUST cause the DDP Stream to be abortively 1069 torn down. 1071 If the Local Peer LLP supports a half-closed LLP Stream, on the 1072 receipt of a LLP graceful teardown request of the DDP Stream, DDP 1073 SHOULD indicate the half-closed state to the ULP, and continue to 1074 process outbound data transfer requests normally. Following this 1075 event, when the Local Peer ULP requests graceful teardown, DDP MUST 1076 indicate to the LLP that it SHOULD perform a graceful close of the 1077 other half of the LLP Stream. 1079 Shah, et. al. Expires March 2007 23 1080 If the Local Peer LLP supports a half-closed LLP Stream, on the 1081 receipt of a ULP graceful half-close teardown request of the DDP 1082 Stream, DDP SHOULD keep data reception enabled on the other half of 1083 the LLP Stream. 1085 6.2.2 DDP Abortive Teardown 1087 As previously mentioned, DDP does not independently terminate a DDP 1088 Stream. Thus any of the following fatal errors on a DDP Stream MUST 1089 cause DDP to indicate to the ULP that a fatal error has occurred: 1091 * Underlying LLP Connection or LLP Stream is lost. 1093 * Underlying LLP reports a fatal error. 1095 * DDP Header has one or more invalid fields. 1097 If the LLP indicates to the ULP that a fatal error has occurred, the 1098 DDP layer SHOULD report the error to the ULP (see Section 7.2, DDP 1099 Error Numbers) and complete all outstanding ULP requests with an 1100 error. If the underlying LLP Stream is still intact, DDP SHOULD 1101 continue to allow the ULP to transfer additional DDP Messages on the 1102 outgoing half connection after the fatal error was indicated to the 1103 ULP. This enables the ULP to transfer an error syndrome to the 1104 Remote Peer. After indicating to the ULP a fatal error has occurred, 1105 the DDP Stream MUST NOT be terminated until the Local Peer ULP 1106 indicates to the DDP layer that the DDP Stream should be abortively 1107 torndown. 1109 Shah, et. al. Expires March 2007 24 1110 7 Error Semantics 1112 All LLP errors reported to DDP SHOULD be passed up to the ULP. 1114 7.1 Errors detected at the Data Sink 1116 For non-zero length Untagged DDP Segments, the DDP Segment MUST be 1117 validated before Placement by verifying: 1119 1. The QN is valid for this stream. 1121 2. The QN and MSN have an associated buffer that allows Placement 1122 of the payload. 1124 Implementers note: DDP implementations SHOULD consider lack of 1125 an associated buffer as a system fault. DDP implementations MAY 1126 try to recover from the system fault using LLP means in a ULP- 1127 transparent way. DDP implementations SHOULD NOT permit system 1128 faults to occur repeatedly or frequently. If there is not an 1129 associated buffer, DDP implementations MAY choose to disable 1130 the stream for the reception and report an error to the ULP at 1131 the Data Sink. 1133 3. The MO falls in the range of legal offsets associated with the 1134 Untagged Buffer. 1136 4. The sum of the DDP Segment payload length and the MO falls in 1137 the range of legal offsets associated with the Untagged Buffer. 1139 5. The Message Sequence Number falls in the range of legal Message 1140 Sequence Numbers, for the queue defined by the QN. The legal 1141 range is defined as being between the MSN value assigned to the 1142 first available buffer for a specific QN and the MSN value 1143 assigned to the last available buffer for a specific QN. 1145 Implementers note: for a typical Queue Number, the lower limit 1146 of the Message Sequence Number is defined by whatever DDP 1147 Messages have already been Completed. The upper limit is 1148 defined by however many message buffers are currently available 1149 for that queue. Both numbers change dynamically as new DDP 1150 Messages are received and Completed, and new buffers are added. 1151 It is up to the ULP to ensure that sufficient buffers are 1152 available to handle the incoming DDP Segments. 1154 For non-zero length Tagged DDP Segments, the segment MUST be 1155 validated before Placement by verifying: 1157 1. The STag is valid for this stream. 1159 2. The STag has an associated buffer that allows Placement of the 1160 payload. 1162 Shah, et. al. Expires March 2007 25 1163 3. The TO falls in the range of legal offsets registered for the 1164 STag. 1166 4. The sum of the DDP Segment payload length and the TO falls in 1167 the range of legal offsets registered for the STag. 1169 5. A 64-bit unsigned sum of the DDP Segment payload length and the 1170 TO does not wrap. 1172 If the DDP layer detects any of the receive errors listed in this 1173 section, it MUST cease placing the remainder of the DDP Segment and 1174 report the error(s) to the ULP. The DDP layer SHOULD include in the 1175 error report the DDP Header, the type of error, and the length of 1176 the DDP segment, if available. DDP MUST silently drop any subsequent 1177 incoming DDP Segments. Since each of these errors represents a 1178 failure of the sending ULP or protocol, DDP SHOULD enable the ULP to 1179 send one additional DDP Message before terminating the DDP Stream. 1181 7.2 DDP Error Numbers 1183 The following error numbers MUST be used when reporting errors to 1184 the ULP. They correspond to the checks enumerated in section 7.1. 1185 Each error is subdivided into a 4-bit Error Type and an 8 bit Error 1186 Code. 1188 Error Error 1189 Type Code Description 1190 ---------------------------------------------------------- 1191 0x0 0x00 Local Catastrophic 1193 0x1 Tagged Buffer Error 1194 0x00 Invalid STag 1195 0x01 Base or bounds violation 1196 0x02 STag not associated with DDP Stream 1197 0x03 TO wrap 1198 0x04 Invalid DDP version 1200 0x2 Untagged Buffer Error 1201 0x01 Invalid QN 1202 0x02 Invalid MSN - no buffer available 1203 0x03 Invalid MSN - MSN range is not valid 1204 0x04 Invalid MO 1205 0x05 DDP Message too long for available buffer 1206 0x06 Invalid DDP version 1208 0x3 Rsvd Reserved for the use by the LLP 1210 Shah, et. al. Expires March 2007 26 1211 8 Security Considerations 1213 This section discusses both protocol-specific considerations and the 1214 implications of using DDP with existing security mechanisms. The 1215 security requirements for the DDP implementation are provided at the 1216 end of the section. A more detailed analysis of the security issues 1217 around the implementation and the use of the DDP can be found in 1218 [RDMASEC]. 1220 The IPsec requirements for RDDP are based on the version of IPsec 1221 specified in RFC 2401 [IPSEC] and related RFCs, as profiled by RFC 1222 3723 [RFC 3723], despite the existence of a newer version of IPsec 1223 specified in RFC 4301 [RFC 4301] and related RFCs. One of the 1224 important early applications of the RDDP protocols is their use with 1225 iSCSI [iSER]; RDDP's IPsec requirements follow those of IPsec in 1226 order to facilitate that usage by allowing a common profile of IPsec 1227 to be used with iSCSI and the RDDP protocols. In the future, RFC 1228 3723 may be updated to the newer version of IPsec, the IPsec 1229 security requirements of any such update should apply uniformly to 1230 iSCSI and the RDDP protocols. 1232 8.1 Protocol-specific Security Considerations 1234 The vulnerabilities of DDP to active third-party interference are no 1235 greater than any other protocol running over transport protocols 1236 such as TCP and SCTP over IP. A third party, by injecting spoofed 1237 packets into the network that are Delivered to a DDP Data Sink, 1238 could launch a variety of attacks that exploit DDP-specific 1239 behavior. Since DDP directly or indirectly exposes memory addresses 1240 on the wire, the Placement information carried in each DDP Segment 1241 must be validated, including invalid STag and octet level 1242 granularity base and bounds check, before any data is Placed. For 1243 example, a third-party adversary could inject random packets that 1244 appear to be valid DDP Segments and corrupt the memory on a DDP Data 1245 Sink. Since DDP is IP transport protocol independent, communication 1246 security mechanisms such as IPsec [IPSEC] may be used to prevent 1247 such attacks. 1249 8.2 Association of an STag and a DDP Stream 1251 There are several mechanisms for associating an STag and a DDP 1252 Stream. Two required mechanisms for this association are a 1253 Protection Domain (PD) association and a DDP Stream association. 1255 Under the Protection Domain (PD) association, a unique Protection 1256 Domain Identifier (PD ID) is created and used locally to associate 1257 an STag with a set of DDP Streams. Under this mechanism, the use of 1258 the STag is only permitted on the DDP Streams that have the same PD 1259 ID as the STag. For an incoming DDP Segment of a Tagged DDP Message 1260 on a DDP Stream, if the PD ID of the DDP Stream is not the same as 1261 the PD ID of the STag targeted by the Tagged DDP Message, then the 1262 DDP Segment is not placed and the DDP layer MUST surface a local 1264 Shah, et. al. Expires March 2007 27 1265 error to the ULP. Note that the PD ID is locally defined, and cannot 1266 be directly manipulated by the Remote Peer. 1268 Under the DDP Stream association, a DDP Stream is identified locally 1269 by a unique DDP Stream identifier (ID). An STag is associated with a 1270 DDP Stream by using a DDP Stream ID. In this case, for an incoming 1271 DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP 1272 Stream ID of the DDP Stream is not the same as the DDP Stream ID of 1273 the STag targeted by the Tagged DDP Message, then the DDP Segment is 1274 not placed and the DDP layer MUST surface a local error to the ULP. 1275 Note that the DDP Stream ID is locally defined, and cannot be 1276 directly manipulated by the Remote Peer. 1278 A ULP SHOULD associate an STag with at least one DDP Stream. DDP 1279 MUST support Protection Domain association and DDP Stream 1280 association mechanisms for associating an STag and a DDP Stream. 1282 8.3 Security Requirements 1284 [RDMASEC] defines the security model and general assumptions for 1285 RDMAP/DDP. This subsection provides the security requirements for 1286 the DDP implementation. For more details on the type of attacks, 1287 type of attackers, trust models, and resource sharing for the DDP 1288 implementation, the reader is referred to [RDMASEC]. 1290 DDP has several mechanisms that deal with a number of attacks. 1291 These attacks include, but are not limited to: 1293 1. Connection to/from an unauthorized or unauthenticated endpoint. 1294 2. Hijacking of a DDP Stream. 1295 3. Attempts to read or write from unauthorized memory regions. 1296 4. Injection of RDMA Messages within a Stream on a multi-user 1297 operating system by another application. 1299 DDP relies on the LLP to establish the LLP Stream over which DDP 1300 Messages will be carried. DDP itself does nothing to authenticate 1301 the validity of the LLP Stream of either of the endpoints. It is the 1302 responsibility of the ULP to validate the LLP Stream. This is highly 1303 desirable due to the nature of DDP. 1305 Hijacking of an DDP Stream would require that the underlying LLP 1306 Stream is hijacked. This would require knowledge of Advertised 1307 buffers in order to directly Place data into a user buffer and is 1308 therefore constrained by the same techniques mentioned to guard 1309 against attempts to read or write from unauthorized memory regions. 1311 DDP does not require a node to open its buffers to arbitrary attacks 1312 over the DDP Stream. It may access ULP memory only to the extent 1313 that the ULP has enabled and authorized it to do so. The STag 1314 access control model is defined in [RDMASEC]. Specific security 1315 operations include: 1317 Shah, et. al. Expires March 2007 28 1318 1. STags are only valid over the exact byte range established by the 1319 ULP. DDP MUST provide a mechanism for the ULP to establish and 1320 revoke the TO range associated with the ULP Buffer referenced by 1321 the STag. 1322 2. STags are only valid for the duration established by the ULP. The 1323 ULP may revoke them at any time, in accordance with its own upper 1324 layer protocol requirements. DDP MUST provide a mechanism for the 1325 ULP to establish and revoke STag validity. 1326 3. DDP MUST provide a mechanism for the ULP to communicate the 1327 association between a STag and a specific DDP Stream. 1328 4. A ULP may only expose memory to remote access to the extent that 1329 it already had access to that memory itself. 1330 5. If an STag is not valid on a DDP Stream, DDP MUST pass the invalid 1331 access attempt to the ULP. The ULP may provide a mechanism for 1332 terminating the DDP Stream. 1334 Further, DDP provides a mechanism that directly Places incoming 1335 payloads in user-mode ULP Buffers. This avoids the risks of prior 1336 solutions that relied upon exposing system buffers for incoming 1337 payloads. 1339 For the DDP implementation, two components MUST be provided: a RDMA 1340 enabled NIC (RNIC) and a Privileged Resource Manager (PRM). 1342 8.3.1 RNIC Requirements 1344 The RNIC MUST implement the DDP wire Protocol and perform the 1345 security semantics described below. 1347 1. An RNIC MUST ensure that a specific DDP Stream in a specific 1348 Protection Domain cannot access an STag in a different 1349 Protection Domain. 1351 2. An RNIC MUST ensure that if an STag is limited in scope to a 1352 single DDP Stream, no other DDP Stream can use the STag. 1354 3. An RNIC MUST ensure that a Remote Peer is not able to access 1355 memory outside of the buffer specified when the STag was 1356 enabled for remote access. 1358 4. An RNIC MUST provide a mechanism for the ULP to establish and 1359 revoke the association of a ULP Buffer to an STag and TO range. 1361 5. An RNIC MUST provide a mechanism for the ULP to establish and 1362 revoke read, write, or read and write access to the ULP Buffer 1363 referenced by an STag. 1365 6. An RNIC MUST ensure that the network interface can no longer 1366 modify an advertised buffer after the ULP revokes remote access 1367 rights for an STag. 1369 Shah, et. al. Expires March 2007 29 1370 7. An RNIC MUST NOT enable firmware to be loaded on the RNIC 1371 directly from an untrusted Local Peer or Remote Peer, unless 1372 the Peer is properly authenticated (by a mechanism outside the 1373 scope of this specification. The mechanism presumably entails 1374 authenticating that the remote ULP has the right to perform the 1375 update), and the update is done via a secure protocol, such as 1376 IPsec. 1378 8.3.2 Privileged Resources Manager Requirement 1380 The PRM MUST implement the security semantics described below. 1382 1. All Non-Privileged ULP interactions with the RNIC Engine that 1383 could affect other ULPs MUST be done using the Privileged 1384 Resource Manager as a proxy. 1386 2. All ULP resource allocation requests for scarce resources MUST 1387 also be done using a Privileged Resource Manager. 1389 3. The Privileged Resource Manager MUST NOT assume different ULPs 1390 share Partial Mutual Trust unless there is a mechanism to 1391 ensure that the ULPs do indeed share partial mutual trust. 1393 4. If Non-Privileged ULPs are supported, the Privileged Resource 1394 Manager MUST verify that the Non-Privileged ULP has the right 1395 to access a specific Data Buffer before allowing an STag for 1396 which the ULP has access rights to be associated with a 1397 specific Data Buffer. 1399 5. The Privileged Resource Manager SHOULD prevent a Local Peer 1400 from allocating more than its fair share of resources. 1401 If an RNIC provides the ability to share receive buffers across 1402 multiple DDP Streams, the combination of the RNIC and the 1403 Privileged Resource Manager MUST be able to detect if the 1404 Remote Peer is attempting to consume more than its fair share 1405 of resources so that the Local Peer can apply countermeasures 1406 to detect and prevent the attack. 1408 8.4 Security Services for DDP 1410 DDP uses IP based network services, therefore, all exchanged DDP 1411 Segments are vulnerable to spoofing, tampering and information 1412 disclosure attacks. If a DDP Stream may be subject to impersonation 1413 attacks, or Stream hijacking attacks, it is highly RECOMMENDED that 1414 the DDP Stream be authenticated, integrity protected, and protected 1415 from replay attacks; it MAY use confidentiality protection to 1416 protect from eavesdropping. 1418 8.4.1 Available Security Services 1420 IPsec can be used to protect against the packet injection attacks 1421 outlined above. Because IPsec is designed to secure arbitrary IP 1423 Shah, et. al. Expires March 2007 30 1424 packet streams, including streams where packets are lost, DDP can 1425 run on top of IPsec without any change. 1427 DDP security may also profit from SSL or TLS security services 1428 provided for TCP or SCTP based ULPs [TLS] as well as from DTLS 1429 [DTLS] security services provided beneath the transport protocol. 1430 See [RDMASEC] for further discussion of these approaches and the 1431 rationale for selection of IPsec security services for the RDDP 1432 protocols. 1434 8.4.2 Requirements for IPsec Services for DDP 1436 IPsec packets are processed (e.g., integrity checked and possibly 1437 decrypted) in the order they are received, and a DDP Data Sink will 1438 process the decrypted DDP Segments contained in these packets in the 1439 same manner as DDP Segments contained in unsecured IP packets. 1441 The IP Storage working group has defined the normative IPsec 1442 requirements for IP Storage [RFC3723]. Portions of this 1443 specification are applicable to the DDP. In particular, a compliant 1444 implementation of IPsec services MUST meet the requirements as 1445 outlined in Section 2.3 of [RFC3723]. Without replicating the 1446 detailed discussion in [RFC3723], this includes the following 1447 requirements: 1449 1. The implementation MUST support IPsec ESP [RFC2406], as well as 1450 the replay protection mechanisms of IPsec. When ESP is 1451 utilized, per-packet data origin authentication, integrity and 1452 replay protection MUST be used. 1454 2. It MUST support ESP in tunnel mode and MAY implement ESP in 1455 transport mode. 1457 3. It MUST support IKE [RFC2409] for peer authentication, 1458 negotiation of security associations, and key management, using 1459 the IPsec DOI [RFC2407]. 1461 4. It MUST NOT interpret the receipt of a IKE delete message as a 1462 reason for tearing down the DDP stream. Since IPsec 1463 acceleration hardware may only be able to handle a limited 1464 number of active IPsec Security Associations (SAs), idle SAs 1465 may be dynamically brought down and a new SA be brought up 1466 again, if activity resumes. 1468 5. It MUST support peer authentication using a pre-shared key, and 1469 MAY support certificate-based peer authentication using digital 1470 signatures. Peer authentication using the public key encryption 1471 methods [RFC2409] SHOULD NOT be used. 1473 6. It MUST support IKE Main Mode and SHOULD support Aggressive 1474 Mode. IKE Main Mode with pre-shared key authentication SHOULD 1476 Shah, et. al. Expires March 2007 31 1477 NOT be used when either of the peers uses a dynamically 1478 assigned IP address. 1480 7. Access to locally stored secret information (pre-shared or 1481 private key for digital signing) must be suitably restricted, 1482 since compromise of the secret information nullifies the 1483 security properties of the IKE/IPsec protocols. 1485 8. It MUST follow the guidelines of Section 2.3.4 of [RFC3723] on 1486 the setting of IKE parameters to achieve a high level of 1487 interoperability without requiring extensive configuration. 1489 Furthermore, implementation and deployment of the IPsec services for 1490 DDP should follow the Security Considerations outlined in Section 5 1491 of [RFC3723]. 1493 Shah, et. al. Expires March 2007 32 1494 9 IANA Considerations 1496 This document requests no direct action from IANA. The following 1497 consideration is listed here as commentary. 1499 If DDP was enabled a priori for a ULP by connecting to a well-known 1500 port, this well-known port would be registered for the DDP with 1501 IANA. The registration of the well-known port will be the 1502 responsibility of the ULP specification. 1504 Shah, et. al. Expires March 2007 33 1505 10 References 1507 10.1 Normative References 1509 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1510 Requirement Levels", BCP 14, RFC 2119, March 1997. 1512 [RFC2406] Kent, S. and Atkinson, R., "IP Encapsulating Security 1513 Payload (ESP)", RFC 2406, November 1998. 1515 [RFC2407] Piper, D., "The Internet IP Security Domain of 1516 Interpretation of ISAKMP", RFC 2407, November 1998. 1518 [RFC2409] Harkins, D. and Carrel, D., "The Internet Key Exchange 1519 (IKE)", RFC 2409, November 1998. 1521 [RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., Travostino, 1522 F., "Securing Block Storage Protocols over IP", RFC 3723, April 1523 2004. 1525 [MPA] Culley, P., Elzur, U., Recio, R., Bailey, S., Carrier, J., 1526 "Marker PDU Aligned Framing for TCP Specification", Internet 1527 Draft draft-ietf-rddp-mpa-04.txt (work in progress), May 2006 1529 [RDMAP] Recio, R., Culley, P., Garcia, D., Hilland, J., "An RDMA 1530 Protocol Specification", Internet Draft draft-ietf-rddp-rdmap- 1531 06.txt (work in progress), June 2006. 1533 [RDMASEC] Pinkerton J. and Deleganes E., "DDP/RDMAP Security", 1534 draft-ietf-rddp-security-10.txt (work in progress), June 2006. 1536 [SCTP] Stewart, R. et al., "Stream Control Transmission Protocol", 1537 RFC 2960, October 2000. 1539 [SCTPDDP] C. Bestler and R. Stewart, "Stream Control Transmission 1540 Protocol (SCTP) Direct Data Placement (DDP) Adaptation", 1541 Internet Draft draft-ietf-rddp-sctp-04.txt (work in progress), 1542 June 2006. 1544 [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, 1545 September 1981. 1547 10.2 Informative References 1549 [RFC 4301] S. Kent and K. Seo, "Security Architecture for the 1550 Internet Protocol", RFC 4301, December 2005. 1552 [DTLS] Rescorla, E. and Modadugu, N., "Datagram Transport Layer 1553 Security", RFC 4347, April 2006. 1555 [IPSEC] Atkinson, R. and Kent, S., "Security Architecture for the 1556 Internet Protocol", RFC 2401, November 1998. 1558 Shah, et. al. Expires March 2007 34 1560 [TLS] Dierks, T. and Rescorla, E., "The Transport Layer Security 1561 (TLS) Protocol Version 1.1", RFC 4346, April 2006. 1563 [iSER] M. Ko, et. al., "iSCSI Extensions for RDMA Specification�, 1564 Internet Draft draft-ietf-ips-iser-05.txt (work in progress), 1565 October 2005. 1567 Shah, et. al. Expires March 2007 35 1568 11 Appendix 1570 11.1 Receive Window sizing 1572 This section provides guidance to LLP implementers. 1574 Reliable, sequenced, LLPs include a mechanism to advertise the 1575 amount of receive buffer space a sender may consume. This is 1576 generally called a "receive window". 1578 DDP allows data to be transferred directly to predefined buffers at 1579 the Data Sink. Accordingly, the LLP receive window size need not be 1580 affected by the reception of a DDP Segment, if that segment is 1581 placed before additional segments arrive. 1583 The LLP implementation SHOULD maintain an advertised receive window 1584 large enough to enable a reasonable number of segments to be 1585 outstanding at one time. The amount to advertise depends on the 1586 desired data rate, and the expected or actual round trip delay 1587 between endpoints. 1589 The amount of actual buffers maintained to "back up" the receive 1590 window is left up to the implementation. This amount will depend on 1591 the rate that DDP Segments can be retired; there may be some cases 1592 where segment processing cannot keep up with the incoming packet 1593 rate. If this occurs, one reasonable way to slow the incoming packet 1594 rate is to reduce the receive window. 1596 Note that the LLP should take care to comply with the applicable 1597 RFCs; for instance, for TCP, receivers are highly discouraged from 1598 "shrinking" the receive window (reducing the right edge of the 1599 window after it has been advertised). 1601 Shah, et. al. Expires March 2007 36 1602 12 Authors' Addresses 1604 Hemal Shah 1605 Broadcom Corporation 1606 16215 Alton Parkway 1607 Irvine, CA. USA 92619-7013 1608 Phone: 949-926-6941 1609 Email: hemal@broadcom.com 1611 James Pinkerton 1612 Microsoft Corporation 1613 One Microsoft Way 1614 Redmond, WA 98052 USA 1615 Phone: +1 (425) 705-5442 1616 Email: jpink@microsoft.com 1618 Renato Recio 1619 IBM Corporation 1620 11501 Burnett Road 1621 Austin, TX 78758 USA 1622 Phone: +1 (512) 838-1365 1623 Email: recio@us.ibm.com 1625 Paul R. Culley 1626 Hewlett-Packard Company 1627 20555 SH 249 1628 Houston, TX 77070-2698 USA 1629 Phone: +1 (281) 514-5543 1630 Email: paul.culley@hp.com 1632 Shah, et. al. Expires March 2007 37 1633 13 Contributors 1635 Many thanks to the following individuals for their contributions. 1637 John Carrier 1638 Adaptec, Inc. 1639 691 S. Milpitas Blvd. 1640 Milpitas, CA 95035 USA 1641 Phone: +1 (360) 378-8526 1642 Email: john_carrier@adaptec.com 1644 Hari Ghadia 1645 Adaptec, Inc. 1646 691 S. Milpitas Blvd., 1647 Milpitas, CA 95035 USA 1648 Phone: +1 (408) 957-5608 1649 Email: hari_ghadia@adaptec.com 1651 Caitlin Bestler 1652 Broadcom Corporation 1653 16215 Alton Parkway 1654 Irvine, California 92619-7013 USA 1655 Phone: +1 (949) 926-6383 1656 Email: caitlinb@Broadcom.com 1658 Uri Elzur 1659 Broadcom Corporation 1660 16215 Alton Parkway 1661 Irvine, California 92619-7013 USA 1662 Phone: +1 (949) 585-6432 1663 Email: uri@Broadcom.com 1665 Mike Penna 1666 Broadcom Corporation 1667 16215 Alton Parkway 1668 Irvine, California 92619-7013 USA 1669 Phone: +1 (949) 926-7149 1670 Email: MPenna@Broadcom.com 1672 Patricia Thaler 1673 Broadcom Corporation 1674 16215 Alton Parkway 1675 Irvine, CA. USA 92619-7013 1676 Phone: +1-949-926-8635 1677 email: pthaler@broadcom.com 1679 Ted Compton 1680 EMC Corporation 1681 Research Triangle Park, NC 27709, USA 1682 Phone: 919-248-6075 1683 Email: compton_ted@emc.com 1685 Shah, et. al. Expires March 2007 38 1686 Jim Wendt 1687 Hewlett-Packard Company 1688 8000 Foothills Boulevard 1689 Roseville, CA 95747-5668 USA 1690 Phone: +1 (916) 785-5198 1691 Email: jim_wendt@hp.com 1693 Mike Krause 1694 Hewlett-Packard Company, 43LN 1695 19410 Homestead Road 1696 Cupertino, CA 95014 USA 1697 Phone: +1 (408) 447-3191 1698 Email: krause@cup.hp.com 1700 Dave Minturn 1701 Intel Corporation 1702 MS JF1-210 1703 5200 North East Elam Young Parkway 1704 Hillsboro, OR 97124 USA 1705 Phone: +1 (503) 712-4106 1706 Email: dave.b.minturn@intel.com 1708 Howard C. Herbert 1709 Intel Corporation 1710 MS CH7-404 1711 5000 West Chandler Blvd. 1712 Chandler, AZ 85226 USA 1713 Phone: +1 (480) 554-3116 1714 Email: howard.c.herbert@intel.com 1716 Tom Talpey 1717 Network Appliance 1718 375 Totten Pond Road 1719 Waltham, MA 02451 USA 1720 Phone: +1 (781) 768-5329 1721 EMail: thomas.talpey@netapp.com 1723 Dwight Barron 1724 Hewlett-Packard Company 1725 20555 SH 249 1726 Houston, TX 77070-2698 USA 1727 Phone: +1 (281) 514-2769 1728 Email: Dwight.Barron@Hp.com 1730 Dave Garcia 1731 Hewlett-Packard Company 1732 19333 Vallco Parkway 1733 Cupertino, Ca. 95014 USA 1734 Phone: +1 (408) 285-6116 1735 Email: dave.garcia@hp.com 1737 Jeff Hilland 1738 Hewlett-Packard Company 1740 Shah, et. al. Expires March 2007 39 1741 20555 SH 249 1742 Houston, Tx. 77070-2698 USA 1743 Phone: +1 (281) 514-9489 1744 Email: jeff.hilland@hp.com 1746 Barry Reinhold 1747 Lamprey Networks 1748 Durham, NH 03824 USA 1749 Phone: +1 (603) 868-8411 1750 Email: bbr@LampreyNetworks.com 1752 Shah, et. al. Expires March 2007 40 1753 14 Intellectual Property Statement 1755 The IETF takes no position regarding the validity or scope of any 1756 Intellectual Property Rights or other rights that might be claimed 1757 to pertain to the implementation or use of the technology described 1758 in this document or the extent to which any license under such 1759 rights might or might not be available; nor does it represent that 1760 it has made any independent effort to identify any such rights. 1761 Information on the procedures with respect to rights in RFC 1762 documents can be found in BCP 78 and BCP 79. 1764 Copies of IPR disclosures made to the IETF Secretariat and any 1765 assurances of licenses to be made available, or the result of an 1766 attempt made to obtain a general license or permission for the use 1767 of such proprietary rights by implementers or users of this 1768 specification can be obtained from the IETF on-line IPR repository 1769 at http://www.ietf.org/ipr. 1771 The IETF invites any interested party to bring to its attention any 1772 copyrights, patents or patent applications, or other proprietary 1773 rights that may cover technology that may be required to implement 1774 this standard. Please address the information to the IETF at ietf- 1775 ipr@ietf.org. 1777 Shah, et. al. Expires March 2007 41 1778 15 Copyright Notice 1780 Copyright (C) The Internet Society (2006). 1782 This document is subject to the rights, licenses and restrictions 1783 contained in BCP 78, and except as set forth therein, the authors 1784 retain all their rights. 1786 This document and the information contained herein are provided on 1787 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 1788 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE 1789 INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR 1790 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 1791 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1792 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1794 Shah, et. al. Expires March 2007 42