idnits 2.17.1 draft-ietf-rddp-arch-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'DAFS' is defined on line 803, but no explicit reference was found in the text == Unused Reference: 'FCVI' is defined on line 808, but no explicit reference was found in the text == Unused Reference: 'IB' is defined on line 813, but no explicit reference was found in the text == Unused Reference: 'MYR' is defined on line 817, but no explicit reference was found in the text == Unused Reference: 'SDP' is defined on line 833, but no explicit reference was found in the text == Unused Reference: 'SRVNET' is defined on line 839, but no explicit reference was found in the text == Unused Reference: 'VI' is defined on line 843, but no explicit reference was found in the text == Outdated reference: A later version (-05) exists of draft-ietf-rddp-problem-statement-03 -- Obsolete informational reference (is this intentional?): RFC 2960 (ref. 'SCTP') (Obsoleted by RFC 4960) Summary: 1 error (**), 0 flaws (~~), 10 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet-Draft Stephen Bailey (Sandburst) 2 Expires: July 2004 Tom Talpey (NetApp) 4 The Architecture of Direct Data Placement (DDP) 5 and Remote Direct Memory Access (RDMA) 6 on Internet Protocols 7 draft-ietf-rddp-arch-04 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance with 12 all provisions of Section 10 of RFC2026. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six 20 months and may be updated, replaced, or obsoleted by other 21 documents at any time. It is inappropriate to use Internet-Drafts 22 as reference material or to cite them other than as "work in 23 progress." 25 The list of current Internet-Drafts can be accessed at 26 http://www.ietf.org/ietf/1id-abstracts.txt 28 The list of Internet-Draft Shadow Directories can be accessed at 29 http://www.ietf.org/shadow.html. 31 Copyright Notice 33 Copyright (C) The Internet Society (2004). All Rights Reserved. 35 Abstract 37 This document defines an abstract architecture for Direct Data 38 Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to 39 run on Internet Protocol-suite transports. This architecture does 40 not necessarily reflect the proper way to implement such protocols, 41 but is, rather, a descriptive tool for defining and understanding 42 the protocols. DDP allows the efficient placement of data into 43 buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA 44 provides the semantics to enable Remote Direct Memory Access 45 between peers in a way consistent with application requirements. 47 Table Of Contents 49 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 50 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . 2 51 1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3 52 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4 53 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4 54 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6 55 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7 56 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10 57 2.2. Remote Direct Memory Access Protocol Architecture . . . 12 58 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 13 59 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16 60 3. Security Considerations . . . . . . . . . . . . . . . . 17 61 4. IANA Considerations . . . . . . . . . . . . . . . . . . 18 62 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 18 63 Informative References . . . . . . . . . . . . . . . . . 18 64 Authors' Addresses . . . . . . . . . . . . . . . . . . . 19 65 Full Copyright Statement . . . . . . . . . . . . . . . . 19 67 1. Introduction 69 This document defines an abstract architecture for Direct Data 70 Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to 71 run on Internet Protocol-suite transports. This architecture does 72 not necessarily reflect the proper way to implement such protocols, 73 but is, rather, a descriptive tool for defining and understanding 74 the protocols. This document uses C language notation as a 75 shorthand to describe the architectural elements of DDP and RDMA 76 protocols. The choice of C notation is not intended to describe 77 concrete protocols or programming interfaces. 79 The first part of the document describes the architecture of DDP 80 protocols, including what assumptions are made about the transports 81 on which DDP is built. The second part describes the architecture 82 of RDMA protocols layered on top of DDP. 84 1.1. Terminology 86 Before introducing the protocols, certain definitions will be 87 useful to guide discussion: 89 o Placement - writing to a data buffer. 91 o Operation - a protocol message, or sequence of messages, which 92 provide a architectural semantic, such as reading or writing 93 of a data buffer. 95 o Delivery - informing any Upper Layer or application that a 96 particular message is available for use. Delivery therefore 97 may be viewed as the "control" signal associated with a unit 98 of data. Note that the order of delivery is defined more 99 strictly than it is for placement. 101 o Completion - informing any Upper Layer or application that a 102 particular operation has finished. A completion, for 103 instance, may require the delivery of several messages, or it 104 may also reflect that some local processing has finished. 106 o Data Sink - the peer on which any placement occurs. 108 o Data Source - the peer from which the placed data originates. 110 o Steering Tag - a "handle" used to identify memory which is the 111 target of placement. A "tagged" message is one which 112 references such a handle. 114 o RDMA Write - an Operation which places data from a local data 115 buffer to a remote data buffer specified by a Steering Tag. 117 o RDMA Read - an Operation which places data to a local data 118 buffer specified by a Steering Tag from a remote data buffer 119 specified by another Steering Tag. 121 o Send - an Operation which places data from a local data buffer 122 to a remote data buffer of the data sink's choice. Sends are 123 therefore "untagged". 125 1.2. DDP and RDMA Protocols 127 The goal of the DDP protocol is to allow the efficient placement of 128 data into buffers designated by protocols layered above DDP (e.g. 129 RDMA). This is described in detail in [ROM]. Efficiency may be 130 characterized by the minimization of the number of transfers of the 131 data over the receiver's system buses. 133 The goal of the RDMA protocol is to provide the semantics to enable 134 Remote Direct Memory Access between peers in a way consistent with 135 application requirements. The RDMA protocol provides facilities 136 immediately useful to existing and future networking, storage, and 137 other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET, 138 VI] 140 The DDP and RDMA protocols work together to achieve their 141 respective goals. DDP provides facilities to safely steer payloads 142 to specific buffers at the Data Sink. RDMA provides facilities to 143 Upper Layers for identifying these buffers, controlling the 144 transfer of data between peers' buffers, supporting authorized 145 bidirectional transfer between buffers, and signalling completion. 146 Upper Layer Protocols that do not require the features of RDMA may 147 be layered directly on top of DDP. 149 The DDP and RDMA protocols are transport independent. The 150 following figure shows the relationship between RDMA, DDP, Upper 151 Layer Protocols and Transport. 153 +--------------------------------------------------+ 154 | Upper Layer Protocol | 155 +---------+------------+---------------------------+ 156 | | | RDMA | 157 | | +---------------------------+ 158 | | DDP | 159 | +----------------------------------------+ 160 | Transport | 161 +--------------------------------------------------+ 163 2. Architecture 165 The Architecture section is presented in two parts: Direct Data 166 Placement Protocol architecture and Remote Direct Memory Access 167 Protocol architecture. 169 2.1. Direct Data Placement (DDP) Protocol Architecture 171 The central idea of general-purpose DDP is that a data sender will 172 supplement the data it sends with placement information that allows 173 the receiver's network interface to place the data directly at its 174 final destination without any copying. DDP can be used to steer 175 received data to its final destination, without requiring layer- 176 specific behavior for each different layer. Data sent with such 177 DDP information is said to be `tagged'. 179 The central component of the DDP architecture is the `buffer', 180 which is an object with beginning and ending addresses, and a 181 method (set()) to set the value of an octet at an address. In many 182 cases, a buffer corresponds directly to a portion of host user 183 memory. However, DDP does not depend on this---a buffer could be a 184 disk file, or anything else that can be viewed as an addressable 185 collection of octets. Abstractly, a buffer provides the interface: 187 typedef struct { 188 const address_t start; 189 const address_t end; 190 void set(address_t a, data_t v); 191 } ddp_buffer_t; 193 address_t 195 a reference to local memory 197 data_t 199 an octet data value. 201 The protocol layering and in-line data flow of DDP is: 203 DDP Client Protocol 204 (e.g. RDMA or Upper Layer Protocol) 205 | ^ 206 untagged messages | | untagged message delivery 207 tagged messages | | tagged message delivery 208 v | 209 DDP+---> data placement 210 ^ 211 | transport messages 212 v 213 Transport 214 (e.g. SCTP, DCCP, framed TCP) 215 ^ 216 | IP datagrams 217 v 218 . . . 220 In addition to in-line data flow, the client protocol registers 221 buffers with DDP, and DDP performs buffer update (set()) operations 222 as a result of receiving tagged messages. 224 DDP messages may be split into multiple, smaller DDP messages, each 225 in a separate transport message. However, if the transport is 226 unreliable or unordered, messages split across transport messages 227 may or may not provide useful behavior, in the same way as 228 splitting arbitrary Upper Layer messages across unreliable or 229 unordered transport messages may or may not provide useful 230 behavior. In other words, the same considerations apply to 231 building client protocols on different types of transports with or 232 without the use of DDP. 234 A DDP message split across transport messages looks like: 236 DDP message: Transport messages: 238 stag=s, offset=o, message 1: 239 notify=y, id=i |type=ddp | 240 message= |stag=s | 241 |aabbccddee|-------. |offset=o | 242 ~ ... ~----. \ |notify=n | 243 |vvwwxxyyzz|-. \ \ |id=? | 244 | \ `--->|aabbccddee| 245 | \ ~ ... ~ 246 | +----->|iijjkkllmm| 247 | | 248 + | message 2: 249 \ | |type=ddp | 250 \ | |stag=s | 251 \ + |offset=o+n| 252 \ \ |notify=y | 253 \ \ |id=i | 254 \ `-->|nnooppqqrr| 255 \ ~ ... ~ 256 `---->|vvwwxxyyzz| 258 Although this picture suggests that DDP information is carried in- 259 line with the message payload, components of the DDP information 260 may also be in transport-specific fields, or derived from 261 transport-specific control information if the transport permits. 263 2.1.1. Transport Operations 265 For the purposes of this architecture, the transport provides: 267 void xpt_send(socket_t s, message_t m); 268 message_t xpt_recv(socket_t s); 269 msize_t xpt_max_msize(socket_t s); 271 socket_t 273 a transport address, including IP addresses, ports and other 274 transport-specific identifiers. 276 message_t 278 a string of octets. 280 msize_t (scalar) 282 a message size. 284 xpt_send(socket_t s, message_t m) 286 send a transport message. 288 xpt_recv(socket_t s) 290 receive a transport message. 292 xpt_max_msize(socket_t s) 294 get the current maximum transport message size. Corresponds, 295 roughly, to the current path Maximum Transfer Unit (PMTU), 296 adjusted by underlying protocol overheads. 298 Real implementations of xpt_send() and xpt_recv() typically return 299 error indications, but that is not relevant to this architecture. 301 2.1.2. DDP Operations 303 The DDP layer provides: 305 void ddp_send(socket_t s, message_t m); 306 void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, 307 ddp_notify_t n); 308 void ddp_post_recv(socket_t s, bdesc_t b); 309 ddp_ind_t ddp_recv(socket_t s); 310 bdesc_t ddp_register(socket_t s, ddp_buffer_t b); 311 void ddp_deregister(bhand_t bh); 312 msizes_t ddp_max_msizes(socket_t s); 314 ddp_addr_t 316 the buffer address portion of a tagged message: 318 typedef struct { 319 stag_t stag; 320 address_t offset; 321 } ddp_addr_t; 323 stag_t (scalar) 324 a Steering Tag. A stag_t identifies the destination buffer 325 for tagged messages. stag_ts are generated when the buffer is 326 registered, communicated to the sender by some client protocol 327 convention and inserted in DDP messages. stag_t values in 328 this DDP architecture are assumed to be completely opaque to 329 the client protocol, and implementation-dependent. However, 330 particular implementations, such as DDP on a multicast 331 transport (see below), may provide the buffer holder some 332 control in selecting stag_ts. 334 ddp_notify_t 336 the notification portion of a DDP message, used to signal that 337 the message represents the final fragment of a multi-segmented 338 DDP message: 340 typedef struct { 341 boolean_t notify; 342 ddp_msg_id_t i; 343 } ddp_notify_t; 345 ddp_msg_id_t (scalar) 347 a DDP message identifier. msg_id_ts are chosen by the DDP 348 message receiver (buffer holder), communicated to the sender 349 by some client protocol convention and inserted in DDP 350 messages. Whether a message reception indication is requested 351 for a DDP message is a matter of client protocol convention. 352 Unlike stag_ts, the structure of msg_id_ts is opaque to DDP, 353 and therefore, completely in the hands of the client protocol. 355 bdesc_t 357 a description of a registered buffer: 359 typedef struct { 360 bhand_t bh; 361 ddp_addr_t a; 362 } bdesc_t; 364 `a.offset' is the starting offset of the registered buffer, 365 which may have no relationship to the `start' or `end' 366 addresses of that buffer. However, particular 367 implementations, such as DDP on a multicast transport (see 368 below), may allow some client protocol control over the 369 starting offset. 371 bhand_t 373 an opaque buffer handle used to deregister a buffer. 375 recv_message_t 377 a description of a completed untagged receive buffer: 379 typedef struct { 380 bdesc_t b; 381 length_t l; 382 } recv_message_t; 384 ddp_ind_t 386 an untagged message, a tagged message reception indication, or 387 a tagged message reception error: 389 typedef union { 390 recv_message_t m; 391 ddp_msg_id_t i; 392 ddp_err_t e; 393 } ddp_ind_t; 395 ddp_err_t 397 indicates an error while receiving a tagged message, typically 398 `offset' out of bounds, or `stag' is not registered to the 399 socket. 401 msizes_t 403 The maximum untagged and tagged messages that fit in a single 404 transport message: 406 typedef struct { 407 msize_t max_untagged; 408 msize_t max_tagged; 409 } msizes_t; 411 ddp_send(socket_t s, message_t m) 413 send an untagged message. 415 ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) 416 send a tagged message to remote buffer address d. 418 ddp_post_recv(socket_t s, bdesc_t b) 420 post a registered buffer to accept a single received untagged 421 message. Each buffer is returned to the caller in a 422 ddp_recv() untagged message reception indication, in the order 423 in which it was posted. The same buffer may be enabled on 424 multiple sockets, receipt of an untagged message into the 425 buffer from any of these sockets unposts the buffer from all 426 sockets. 428 ddp_recv(socket_t s) 430 get the next received untagged message, tagged message 431 reception indication, or tagged message error. 433 ddp_register(socket_t s, ddp_buffer_t b) 435 register a buffer for DDP on a socket. The same buffer may be 436 registered multiple times on the same or different sockets. 437 The same buffer registered on different sockets may result in 438 a common registration. Different buffers may also refer to 439 portions of the same underlying addressable object (buffer 440 aliasing). 442 ddp_deregister(bhand_t bh) 444 remove a registration from a buffer. 446 ddp_max_msizes(socket_t s) 448 get the current maximum untagged and tagged message sizes that 449 will fit in a single transport message. 451 2.1.3. Transport Characteristics In DDP 453 Certain characteristics of the transport on which DDP is mapped 454 determine the nature of the service provided to client protocols. 455 Fundamentally, the characteristics of the transport will not be 456 changed by the presence of DDP. The choice of transport is 457 therefore driven not by DDP, but by the requirements of the Upper 458 Layer, and employing the DDP service. 460 Specifically, transports are: 462 o reliable or unreliable, 463 o ordered or unordered, 465 o single source or multisource, 467 o single destination or multidestination (multicast or anycast). 469 Some transports support several combinations of these 470 characteristics. For example, SCTP [SCTP] is reliable, single 471 source, single destination (point-to-point) and supports both 472 ordered and unordered modes. 474 DDP messages carried by transport are framed for processing by the 475 receiver, and may be further protected for integrity or privacy in 476 accordance with the transport capabilities. DDP does not provide 477 such functions. 479 In general, transport characteristics equally affect transport and 480 DDP message delivery. However, there are several issues specific 481 to DDP messages. 483 A key component of DDP is how the following operations on the 484 receiving side are ordered among themselves, and how they relate to 485 corresponding operations on the sending side: 487 o set()s, 489 o untagged message reception indications, and 491 o tagged message reception indications. 493 These relationships depend upon the characteristics of the 494 underlying transport in a way which is defined by the DDP protocol. 495 For example, if the transport is unreliable and unordered, the DDP 496 protocol might specify that the client protocol is subject to the 497 consequences of transport messages being lost or duplicated, rather 498 than requiring different characteristics be presented to the client 499 protocol. 501 Multidestination data delivery is the other transport 502 characteristic which may require specific consideration in a DDP 503 protocol. As mentioned above, the basic DDP model assumes that 504 buffer address values returned by ddp_register() are opaque to the 505 client protocol, and can be implementation dependent. The most 506 natural way to map DDP to a multidestination transport is to 507 require all receivers produce the same buffer address when 508 registering a multidestination destination buffer. Restriction of 509 the DDP model to accommodate multiple destinations involves 510 engineering tradeoffs comparable to those of providing non-DDP 511 multidestination transport capability. 513 The same buffer may be enabled by ddp_post_recv() on multiple 514 sockets. In this case the ddp_recv() untagged message reception 515 indication may be provided on a different socket from that on which 516 the buffer was posted. Such indications are not ordered among 517 multiple DDP sockets. 519 When multiple sockets reference an untagged message reception 520 buffer, local interfaces are responsible for managing the 521 mechanisms of allocating posted buffers to received untagged 522 messages, the handling of received untagged messages when no buffer 523 is available, and of resource management among multiple sockets. 524 Where underprovisioning of buffers on multiple sockets is allowed, 525 mechanisms should be provided to manage buffer consumption on a 526 per-socket or group of related sockets basis. 528 Architecturally, therefore, DDP is a flexible and general paradigm 529 which may be applied to any variety of transports. Implementations 530 of DDP may, however, adapt themselves to these differences in ways 531 appropriate to each transport. In all cases the layering of DDP 532 must continue to express the transport's underlying 533 characteristics. 535 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture 537 Remote Direct Memory Access (RDMA) extends the capabilities of DDP 538 with two primary functions. 540 First, it adds the ability to read from buffers registered to a 541 socket (RDMA Read). This allows a client protocol to perform 542 arbitrary, bidirectional data movement without involving the remote 543 client. When RDMA is implemented in hardware, arbitrary data 544 movement can be performed without involving the remote host CPU at 545 all. 547 In addition, RDMA specifies a transport-independent untagged 548 message service (Send) with characteristics which are both very 549 efficient to implement in hardware, and convenient for client 550 protocols. 552 The RDMA architecture is patterned after the traditional model for 553 device programming, where the client requests an operation using 554 Send-like actions (programmed I/O), the server performs the 555 necessary data transfers for the operation (DMA reads and writes), 556 and notifies the client of completion. The programmed I/O+DMA 557 model efficiently supports a high degree of concurrency and 558 flexibility for both the client and server, even when operations 559 have a wide range of intrinsic latencies. 561 RDMA is layered as a client protocol on top of DDP: 563 Client Protocol 564 | ^ 565 Sends | | Send reception indications 566 RDMA Read Requests | | RDMA Read Completion indications 567 RDMA Writes | | RDMA Write Completion indications 568 v | 569 RDMA 570 | ^ 571 untagged messages | | untagged message delivery 572 tagged messages | | tagged message delivery 573 v | 574 DDP+---> data placement 575 ^ 576 | transport messages 577 v 578 . . . 580 In addition to in-line data flow, read (get()) and update (set()) 581 operations are performed on buffers registered with RDMA as a 582 result of RDMA Read Requests and RDMA Writes, respectively. 584 An RDMA `buffer' extends a DDP buffer with a get() operation that 585 retrieves the value of the octet at address `a': 587 typedef struct { 588 const address_t start; 589 const address_t end; 590 void set(address_t a, data_t v); 591 data_t get(address_t a); 592 } rdma_buffer_t; 594 2.2.1. RDMA Operations 596 The RDMA layer provides: 598 void rdma_send(socket_t s, message_t m); 599 void rdma_write(socket_t s, message_t m, ddp_addr_t d, 600 rdma_notify_t n); 601 void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); 602 void rdma_post_recv(socket_t s, bdesc_t b); 603 rdma_ind_t rdma_recv(socket_t s); 604 bdesc_t rdma_register(socket_t s, rdma_buffer_t b, 605 bmode_t mode); 606 void rdma_deregister(bhand_t bh); 607 msizes_t rdma_max_msizes(socket_t s); 609 Although, for clarity, these data transfer interfaces are 610 synchronous, rdma_read() and possibly rdma_send() (in the presence 611 of Send flow control), can require an arbitrary amount of time to 612 complete. To express the full concurrency and interleaving of RDMA 613 data transfer, these interfaces should also be reentrant. For 614 example, a client protocol may perform an rdma_send(), while an 615 rdma_read() operation is in progress. 617 rdma_notify_t 619 RDMA Write notification information, used to signal that the 620 message represents the final fragment of a multi-segmented 621 RDMA message: 623 typedef struct { 624 boolean_t notify; 625 rdma_write_id_t i; 626 } rdma_notify_t; 628 identical in function to ddp_notify_t, except that the type 629 rdma_write_id_t may not be equivalent to ddp_msg_id_t. 631 rdma_write_id_t (scalar) 633 an RDMA Write identifier. 635 rdma_ind_t 637 a Send message, or an RDMA error: 639 typedef union { 640 recv_message_t m; 641 rdma_err_t e; 642 } rdma_ind_t; 644 rdma_err_t 646 an RDMA protocol error indication. RDMA errors include buffer 647 addressing errors corresponding to ddp_err_ts, and buffer 648 protection violations (e.g. RDMA Writing a buffer only 649 registered for reading). 651 bmode_t 653 buffer registration mode (permissions). Any combination of 654 permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE) 655 operations. 657 rdma_send(socket_t s, message_t m) 659 send a message, delivering it to the next untagged RDMA buffer 660 at the remote peer. 662 rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) 664 RDMA Write to remote buffer address d. 666 rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d) 668 RDMA Read l octets from remote buffer address s to local 669 buffer address d. 671 rdma_post_recv(socket_t s, bdesc_t b) 673 post a registered buffer to accept a single Send message, to 674 be filled and returned in-order to a subsequent caller of 675 rdma_recv(). As with DDP, buffers may be enabled on multiple 676 sockets, in which case ordering guarantees are relaxed. Also 677 as with DDP, local interfaces must manage the mechanisms of 678 allocation and management of buffers posted to multiple 679 sockets. 681 rdma_recv(socket_t s); 683 get the next received Send message, RDMA Write completion 684 identifier, or RDMA error. 686 rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) 688 register a buffer for RDMA on a socket (for read access, write 689 access or both). As with DDP, the same buffer may be 690 registered multiple times on the same or different sockets, 691 and different buffers may refer to portions of the same 692 underlying addressable object. 694 rdma_deregister(bhand_t bh) 696 remove a registration from a buffer. 698 rdma_max_msizes(socket_t s) 700 get the current maximum Send (max_untagged) and RDMA Read or 701 Write (max_tagged) operations that will fit in a single 702 transport message. The values returned by rdma_max_msizes() 703 are closely related to the values returned by 704 ddp_max_msizes(), but may not be equal. 706 2.2.2. Transport Characteristics In RDMA 708 As with DDP, RDMA can be used on transports with a variety of 709 different characteristics that manifest themselves directly in the 710 service provided by RDMA. Also as with DDP, the fundamental 711 characteristics of the transport will not be changed by the 712 presence of RDMA. 714 Like DDP, an RDMA protocol must specify how: 716 o set()s, 718 o get()s, 720 o Send messages, and 722 o RDMA Read completions 724 are ordered among themselves and how they relate to corresponding 725 operations on the remote peer(s). These relationships are likely 726 to be a function of the underlying transport characteristics. 728 There are some additional characteristics of RDMA which may 729 translate poorly to unreliable or multipoint transports due to 730 attendant complexities in managing endpoint state: 732 o Send flow control 734 o RDMA Read 736 These difficulties can be overcome by placing restrictions on the 737 service provided by RDMA. However, many RDMA clients, especially 738 those that separate data transfer and application logic concerns, 739 are likely to depend upon capabilities only provided by RDMA on a 740 point-to-point, reliable transport. In other words, many potential 741 Upper Layers which might avail themselves of RDMA services are 742 naturally already biased toward these transport classes. 744 3. Security Considerations 746 Fundamentally, the DDP and RDMA protocols should not introduce 747 additional vulnerabilities. They are intermediate protocols and so 748 should not perform or require functions such as authorization, 749 which are the domain of Upper Layers. However, the DDP and RDMA 750 protocols should allow mapping by strict Upper Layers which are not 751 permissive of new vulnerabilities -- DDP and RDMAP implementations 752 should be prohibited from `cutting corners' that create new 753 vulnerabilities. Implementations must ensure that only `supplied' 754 resources (i.e. buffers) can be manipulated by DDP or RDMAP 755 messages. 757 System integrity must be maintained in any RDMA solution. 758 Mechanisms must be specified to prevent RDMA or DDP operations from 759 impairing system integrity. For example, threats can include 760 potential buffer reuse or buffer overflow, and are not merely a 761 security issue. Even trusted peers must not be allowed to damage 762 local integrity. Any DDP and RDMA protocol must address the issue 763 of giving end-systems and applications the capabilities to offer 764 protection from such compromises. 766 Because a Steering Tag exports access to a memory region, one 767 critical aspect of security is the scope of this access. It must 768 be possible to individually control specific attributes of the 769 access provided by a Steering Tag, including remote read access, 770 remote write access, and others that might be identified. DDP and 771 RDMA specifications must provide both implementation requirements 772 relevant to this issue, and guidelines to assist implementors in 773 making the appropriate design decisions. 775 The use of DDP and RDMA on a transport connection may interact with 776 any security mechanism, and vice-versa. For example, if the 777 security mechanism is implemented above the transport layer, the 778 DDP and RDMA headers may not be protected. Such a layering may 779 therefore be inappropriate, depending on requirements. Or, when 780 TLS is employed, it may not be possible for DDP and RDMA to process 781 segments out of order, due to the in-order requirement of TLS. 782 These interactions should be well explored. 784 Resource issues leading to denial-of-service attacks, overwrites 785 and other concurrent operations, the ordering of completions as 786 required by the RDMA protocol, and the granularity of transfer are 787 all within the required scope of any security analysis of RDMA and 788 DDP. 790 4. IANA Considerations 792 IANA considerations are not addressed in by this document. Any 793 IANA considerations resulting from the use of DDP or RDMA must be 794 addressed in the relevant standards. 796 5. Acknowledgements 798 The authors wish to acknowledge the valuable contributions of 799 Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow. 801 6. Informative References 803 [DAFS] 804 DAFS Collaborative, "Direct Access File System Specification 805 v1.0", September 2001, available from 806 http://www.dafscollaborative.org 808 [FCVI] 809 ANSI Technical Committee T11, "Fibre Channel Standard Virtual 810 Interface Architecture Mapping", ANSI/NCITS 357-2001, March 811 2001, available from http://www.t11.org/t11/stat.nsf/fcproj 813 [IB] InfiniBand Trade Association, "InfiniBand Architecture 814 Specification Volumes 1 and 2", Release 1.1, November 2002, 815 available from http://www.infinibandta.org/specs 817 [MYR] 818 VMEbus International Trade Association, "Myrinet on VME 819 Protocol Specification", ANSI/VITA 26-1998, August 1998, 820 available from http://www.myri.com/open-specs 822 [ROM] 823 A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP 824 Problem Statement", draft-ietf-rddp-problem-statement-03, Work 825 in Progress, January 2004 826 RFC Editor note: Replace problem statement draft-ietf- name, status and 827 date with appropriate reference when assigned. 829 [SCTP] 830 R. Stewart et al., "Stream Transmission Control Protocol", RFC 831 2960, Standards Track 833 [SDP] 834 InfiniBand Trade Association, "Sockets Direct Protocol v1.0", 835 Annex A of InfiniBand Architecture Specification Volume 1, 836 Release 1.1, November 2002, available from 837 http://www.infinibandta.org/specs 839 [SRVNET] 840 R. Horst, "TNet: A reliable system area network", IEEE Micro, 841 pp. 37-45, February 1995 843 [VI] Compaq Computer Corp., Intel Corporation and Microsoft 844 Corporation, "Virtual Interface Architecture Specification 845 Version 1.0", December 1997, available from 846 http://www.vidf.org/info/04standards.html 848 Authors' Addresses 850 Stephen Bailey 851 Sandburst Corporation 852 600 Federal Street 853 Andover, MA 01810 USA 854 USA 856 Phone: +1 978 689 1614 857 Email: steph@sandburst.com 859 Tom Talpey 860 Network Appliance 861 375 Totten Pond Road 862 Waltham, MA 02451 USA 864 Phone: +1 781 768 5329 865 Email: thomas.talpey@netapp.com 867 Full Copyright Statement 869 Copyright (C) The Internet Society (2004). All Rights Reserved. 871 This document and translations of it may be copied and furnished to 872 others, and derivative works that comment on or otherwise explain 873 it or assist in its implementation may be prepared, copied, 874 published and distributed, in whole or in part, without restriction 875 of any kind, provided that the above copyright notice and this 876 paragraph are included on all such copies and derivative works. 877 However, this document itself may not be modified in any way, such 878 as by removing the copyright notice or references to the Internet 879 Society or other Internet organizations, except as needed for the 880 purpose of developing Internet standards in which case the 881 procedures for copyrights defined in the Internet Standards process 882 must be followed, or as required to translate it into languages 883 other than English. 885 The limited permissions granted above are perpetual and will not be 886 revoked by the Internet Society or its successors or assigns. 888 This document and the information contained herein is provided on 889 an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET 890 ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR 891 IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 892 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 893 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.