idnits 2.17.1 draft-ietf-rddp-arch-07.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 15. -- Found old boilerplate from RFC 3978, Section 5.5 on line 977. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 987. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 994. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1000. ** Found boilerplate matching RFC 3978, Section 5.4, paragraph 1 (on line 968), which is fine, but *also* found old RFC 2026, Section 10.4C, paragraph 1 text on line 36. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** The document seems to lack an RFC 3978 Section 5.4 Reference to BCP 78 -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- Couldn't find a document date in the document -- date freshness check skipped. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Unused Reference: 'DAFS' is defined on line 901, but no explicit reference was found in the text == Unused Reference: 'FCVI' is defined on line 906, but no explicit reference was found in the text == Unused Reference: 'IB' is defined on line 911, but no explicit reference was found in the text == Unused Reference: 'MYR' is defined on line 915, but no explicit reference was found in the text == Unused Reference: 'SDP' is defined on line 929, but no explicit reference was found in the text == Unused Reference: 'SRVNET' is defined on line 935, but no explicit reference was found in the text == Unused Reference: 'VI' is defined on line 939, but no explicit reference was found in the text -- Obsolete informational reference (is this intentional?): RFC 2960 (ref. 'SCTP') (Obsoleted by RFC 4960) Summary: 7 errors (**), 0 flaws (~~), 9 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet-Draft Stephen Bailey (Sandburst) 3 Expires: August 2005 Tom Talpey (NetApp) 5 The Architecture of Direct Data Placement (DDP) 6 and Remote Direct Memory Access (RDMA) 7 on Internet Protocols 8 draft-ietf-rddp-arch-07 10 Status of this Memo 12 By submitting this Internet-Draft, I certify that any applicable 13 patent or other IPR claims of which I am aware have been disclosed, 14 or will be disclosed, and any of which I become aware will be 15 disclosed, in accordance with RFC 3668. 17 Internet-Drafts are working documents of the Internet Engineering 18 Task Force (IETF), its areas, and its working groups. Note that 19 other groups may also distribute working documents as Internet- 20 Drafts. 22 Internet-Drafts are draft documents valid for a maximum of six 23 months and may be updated, replaced, or obsoleted by other 24 documents at any time. It is inappropriate to use Internet-Drafts 25 as reference material or to cite them other than as "work in 26 progress." 28 The list of current Internet-Drafts can be accessed at 29 http://www.ietf.org/ietf/1id-abstracts.txt 31 The list of Internet-Draft Shadow Directories can be accessed at 32 http://www.ietf.org/shadow.html 34 Copyright Notice 36 Copyright (C) The Internet Society (2005). All Rights Reserved. 38 Abstract 40 This document defines an abstract architecture for Direct Data 41 Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to 42 run on Internet Protocol-suite transports. This architecture does 43 not necessarily reflect the proper way to implement such protocols, 44 but is, rather, a descriptive tool for defining and understanding 45 the protocols. DDP allows the efficient placement of data into 46 buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA 47 provides the semantics to enable Remote Direct Memory Access 48 between peers in a way consistent with application requirements. 50 Table Of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2 53 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . 2 54 1.2. DDP and RDMA Protocols . . . . . . . . . . . . . . . . . 3 55 2. Architecture . . . . . . . . . . . . . . . . . . . . . . 4 56 2.1. Direct Data Placement (DDP) Protocol Architecture . . . 4 57 2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 6 58 2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 7 59 2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10 60 2.2. Remote Direct Memory Access Protocol Architecture . . . 12 61 2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 14 62 2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 16 63 3. Security Considerations . . . . . . . . . . . . . . . . 17 64 3.1. Security Services . . . . . . . . . . . . . . . . . . . 18 65 3.2. Error Considerations . . . . . . . . . . . . . . . . . . 19 66 4. IANA Considerations . . . . . . . . . . . . . . . . . . 19 67 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 20 68 Informative References . . . . . . . . . . . . . . . . . 20 69 Authors' Addresses . . . . . . . . . . . . . . . . . . . 21 70 Full Copyright Statement . . . . . . . . . . . . . . . . 21 72 1. Introduction 74 This document defines an abstract architecture for Direct Data 75 Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to 76 run on Internet Protocol-suite transports. This architecture does 77 not necessarily reflect the proper way to implement such protocols, 78 but is, rather, a descriptive tool for defining and understanding 79 the protocols. This document uses C language notation as a 80 shorthand to describe the architectural elements of DDP and RDMA 81 protocols. The choice of C notation is not intended to describe 82 concrete protocols or programming interfaces. 84 The first part of the document describes the architecture of DDP 85 protocols, including what assumptions are made about the transports 86 on which DDP is built. The second part describes the architecture 87 of RDMA protocols layered on top of DDP. 89 1.1. Terminology 91 Before introducing the protocols, certain definitions will be 92 useful to guide discussion: 94 o Placement - writing to a data buffer. 96 o Operation - a protocol message, or sequence of messages, which 97 provide a architectural semantic, such as reading or writing 98 of a data buffer. 100 o Delivery - informing any Upper Layer or application that a 101 particular message is available for use. Delivery therefore 102 may be viewed as the "control" signal associated with a unit 103 of data. Note that the order of delivery is defined more 104 strictly than it is for placement. 106 o Completion - informing any Upper Layer or application that a 107 particular operation has finished. A completion, for 108 instance, may require the delivery of several messages, or it 109 may also reflect that some local processing has finished. 111 o Data Sink - the peer on which any placement occurs. 113 o Data Source - the peer from which the placed data originates. 115 o Steering Tag - a "handle" used to identify the buffer which is 116 the target of placement. A "tagged" message is one which 117 references such a handle. 119 o RDMA Write - an Operation which places data from a local data 120 buffer to a remote data buffer specified by a Steering Tag. 122 o RDMA Read - an Operation which places data to a local data 123 buffer specified by a Steering Tag from a remote data buffer 124 specified by another Steering Tag. 126 o Send - an Operation which places data from a local data buffer 127 to a remote data buffer of the data sink's choice. Sends are 128 therefore "untagged". 130 1.2. DDP and RDMA Protocols 132 The goal of the DDP protocol is to allow the efficient placement of 133 data into buffers designated by protocols layered above DDP (e.g. 134 RDMA). This is described in detail in [ROM]. Efficiency may be 135 characterized by the minimization of the number of transfers of the 136 data over the receiver's system buses. 138 The goal of the RDMA protocol is to provide the semantics to enable 139 Remote Direct Memory Access between peers in a way consistent with 140 application requirements. The RDMA protocol provides facilities 141 immediately useful to existing and future networking, storage, and 142 other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET, 143 VI] 144 The DDP and RDMA protocols work together to achieve their 145 respective goals. DDP provides facilities to safely steer payloads 146 to specific buffers at the Data Sink. RDMA provides facilities to 147 Upper Layers for identifying these buffers, controlling the 148 transfer of data between peers' buffers, supporting authorized 149 bidirectional transfer between buffers, and signalling completion. 150 Upper Layer Protocols that do not require the features of RDMA may 151 be layered directly on top of DDP. 153 The DDP and RDMA protocols are transport independent. The 154 following figure shows the relationship between RDMA, DDP, Upper 155 Layer Protocols and Transport. 157 +--------------------------------------------------+ 158 | Upper Layer Protocol | 159 +---------+------------+---------------------------+ 160 | | | RDMA | 161 | | +---------------------------+ 162 | | DDP | 163 | +----------------------------------------+ 164 | Transport | 165 +--------------------------------------------------+ 167 2. Architecture 169 The Architecture section is presented in two parts: Direct Data 170 Placement Protocol architecture and Remote Direct Memory Access 171 Protocol architecture. 173 2.1. Direct Data Placement (DDP) Protocol Architecture 175 The central idea of general-purpose DDP is that a data sender will 176 supplement the data it sends with placement information that allows 177 the receiver's network interface to place the data directly at its 178 final destination without any copying. DDP can be used to steer 179 received data to its final destination, without requiring layer- 180 specific behavior for each different layer. Data sent with such 181 DDP information is said to be `tagged'. 183 The central component of the DDP architecture is the `buffer', 184 which is an object with beginning and ending addresses, and a 185 method (set()) to set the value of an octet at an address. In many 186 cases, a buffer corresponds directly to a portion of host user 187 memory. However, DDP does not depend on this---a buffer could be a 188 disk file, or anything else that can be viewed as an addressable 189 collection of octets. Abstractly, a buffer provides the interface: 191 typedef struct { 192 const address_t start; 193 const address_t end; 194 void set(address_t a, data_t v); 195 } ddp_buffer_t; 197 address_t 199 a reference to local memory 201 data_t 203 an octet data value. 205 The protocol layering and in-line data flow of DDP is: 207 DDP Client Protocol 208 (e.g. RDMA or Upper Layer Protocol) 209 | ^ 210 untagged messages | | untagged message delivery 211 tagged messages | | tagged message delivery 212 v | 213 DDP+---> data placement 214 ^ 215 | transport messages 216 v 217 Transport 218 (e.g. SCTP, DCCP, framed TCP) 219 ^ 220 | IP datagrams 221 v 222 . . . 224 In addition to in-line data flow, the client protocol registers 225 buffers with DDP, and DDP performs buffer update (set()) operations 226 as a result of receiving tagged messages. 228 DDP messages may be split into multiple, smaller DDP messages, each 229 in a separate transport message. However, if the transport is 230 unreliable or unordered, messages split across transport messages 231 may or may not provide useful behavior, in the same way as 232 splitting arbitrary Upper Layer messages across unreliable or 233 unordered transport messages may or may not provide useful 234 behavior. In other words, the same considerations apply to 235 building client protocols on different types of transports with or 236 without the use of DDP. 238 A DDP message split across transport messages looks like: 240 DDP message: Transport messages: 242 stag=s, offset=o, message 1: 243 notify=y, id=i |type=ddp | 244 message= |stag=s | 245 |aabbccddee|-------. |offset=o | 246 ~ ... ~----. \ |notify=n | 247 |vvwwxxyyzz|-. \ \ |id=? | 248 | \ `--->|aabbccddee| 249 | \ ~ ... ~ 250 | +----->|iijjkkllmm| 251 | | 252 + | message 2: 253 \ | |type=ddp | 254 \ | |stag=s | 255 \ + |offset=o+n| 256 \ \ |notify=y | 257 \ \ |id=i | 258 \ `-->|nnooppqqrr| 259 \ ~ ... ~ 260 `---->|vvwwxxyyzz| 262 Although this picture suggests that DDP information is carried in- 263 line with the message payload, components of the DDP information 264 may also be in transport-specific fields, or derived from 265 transport-specific control information if the transport permits. 267 2.1.1. Transport Operations 269 For the purposes of this architecture, the transport provides: 271 void xpt_send(socket_t s, message_t m); 272 message_t xpt_recv(socket_t s); 273 msize_t xpt_max_msize(socket_t s); 275 socket_t 277 a transport address, including IP addresses, ports and other 278 transport-specific identifiers. 280 message_t 282 a string of octets. 284 msize_t (scalar) 286 a message size. 288 xpt_send(socket_t s, message_t m) 290 send a transport message. 292 xpt_recv(socket_t s) 294 receive a transport message. 296 xpt_max_msize(socket_t s) 298 get the current maximum transport message size. Corresponds, 299 roughly, to the current path Maximum Transfer Unit (PMTU), 300 adjusted by underlying protocol overheads. 302 Real implementations of xpt_send() and xpt_recv() typically return 303 error indications, but that is not relevant to this architecture. 305 2.1.2. DDP Operations 307 The DDP layer provides: 309 void ddp_send(socket_t s, message_t m); 310 void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, 311 ddp_notify_t n); 312 void ddp_post_recv(socket_t s, bdesc_t b); 313 ddp_ind_t ddp_recv(socket_t s); 314 bdesc_t ddp_register(socket_t s, ddp_buffer_t b); 315 void ddp_deregister(bhand_t bh); 316 msizes_t ddp_max_msizes(socket_t s); 318 ddp_addr_t 320 the buffer address portion of a tagged message: 322 typedef struct { 323 stag_t stag; 324 address_t offset; 325 } ddp_addr_t; 327 stag_t (scalar) 328 a Steering Tag. A stag_t identifies the destination buffer 329 for tagged messages. stag_ts are generated when the buffer is 330 registered, communicated to the sender by some client protocol 331 convention and inserted in DDP messages. stag_t values in 332 this DDP architecture are assumed to be completely opaque to 333 the client protocol, and implementation-dependent. However, 334 particular implementations, such as DDP on a multicast 335 transport (see below), may provide the buffer holder some 336 control in selecting stag_ts. 338 ddp_notify_t 340 the notification portion of a DDP message, used to signal that 341 the message represents the final fragment of a multi-segmented 342 DDP message: 344 typedef struct { 345 boolean_t notify; 346 ddp_msg_id_t i; 347 } ddp_notify_t; 349 ddp_msg_id_t (scalar) 351 a DDP message identifier. msg_id_ts are chosen by the DDP 352 message receiver (buffer holder), communicated to the sender 353 by some client protocol convention and inserted in DDP 354 messages. Whether a message reception indication is requested 355 for a DDP message is a matter of client protocol convention. 356 Unlike stag_ts, the structure of msg_id_ts is opaque to DDP, 357 and therefore, completely in the hands of the client protocol. 359 bdesc_t 361 a description of a registered buffer: 363 typedef struct { 364 bhand_t bh; 365 ddp_addr_t a; 366 } bdesc_t; 368 `a.offset' is the starting offset of the registered buffer, 369 which may have no relationship to the `start' or `end' 370 addresses of that buffer. However, particular 371 implementations, such as DDP on a multicast transport (see 372 below), may allow some client protocol control over the 373 starting offset. 375 bhand_t 377 an opaque buffer handle used to deregister a buffer. 379 recv_message_t 381 a description of a completed untagged receive buffer: 383 typedef struct { 384 bdesc_t b; 385 length_t l; 386 } recv_message_t; 388 ddp_ind_t 390 an untagged message, a tagged message reception indication, or 391 a tagged message reception error: 393 typedef union { 394 recv_message_t m; 395 ddp_msg_id_t i; 396 ddp_err_t e; 397 } ddp_ind_t; 399 ddp_err_t 401 indicates an error while receiving a tagged message, typically 402 `offset' out of bounds, or `stag' is not registered to the 403 socket. 405 msizes_t 407 The maximum untagged and tagged messages that fit in a single 408 transport message: 410 typedef struct { 411 msize_t max_untagged; 412 msize_t max_tagged; 413 } msizes_t; 415 ddp_send(socket_t s, message_t m) 417 send an untagged message. 419 ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n) 420 send a tagged message to remote buffer address d. 422 ddp_post_recv(socket_t s, bdesc_t b) 424 post a registered buffer to accept a single received untagged 425 message. Each buffer is returned to the caller in a 426 ddp_recv() untagged message reception indication, in the order 427 in which it was posted. The same buffer may be enabled on 428 multiple sockets, receipt of an untagged message into the 429 buffer from any of these sockets unposts the buffer from all 430 sockets. 432 ddp_recv(socket_t s) 434 get the next received untagged message, tagged message 435 reception indication, or tagged message error. 437 ddp_register(socket_t s, ddp_buffer_t b) 439 register a buffer for DDP on a socket. The same buffer may be 440 registered multiple times on the same or different sockets. 441 The same buffer registered on different sockets may result in 442 a common registration. Different buffers may also refer to 443 portions of the same underlying addressable object (buffer 444 aliasing). 446 ddp_deregister(bhand_t bh) 448 remove a registration from a buffer. 450 ddp_max_msizes(socket_t s) 452 get the current maximum untagged and tagged message sizes that 453 will fit in a single transport message. 455 2.1.3. Transport Characteristics In DDP 457 Certain characteristics of the transport on which DDP is mapped 458 determine the nature of the service provided to client protocols. 459 Fundamentally, the characteristics of the transport will not be 460 changed by the presence of DDP. The choice of transport is 461 therefore driven not by DDP, but by the requirements of the Upper 462 Layer, and employing the DDP service. 464 Specifically, transports are: 466 o reliable or unreliable, 467 o ordered or unordered, 469 o single source or multisource, 471 o single destination or multidestination (multicast or anycast). 473 Some transports support several combinations of these 474 characteristics. For example, SCTP [SCTP] is reliable, single 475 source, single destination (point-to-point) and supports both 476 ordered and unordered modes. 478 DDP messages carried by transport are framed for processing by the 479 receiver, and may be further protected for integrity or privacy in 480 accordance with the transport capabilities. DDP does not provide 481 such functions. 483 In general, transport characteristics equally affect transport and 484 DDP message delivery. However, there are several issues specific 485 to DDP messages. 487 A key component of DDP is how the following operations on the 488 receiving side are ordered among themselves, and how they relate to 489 corresponding operations on the sending side: 491 o set()s, 493 o untagged message reception indications, and 495 o tagged message reception indications. 497 These relationships depend upon the characteristics of the 498 underlying transport in a way which is defined by the DDP protocol. 499 For example, if the transport is unreliable and unordered, the DDP 500 protocol might specify that the client protocol is subject to the 501 consequences of transport messages being lost or duplicated, rather 502 than requiring different characteristics be presented to the client 503 protocol. 505 Buffer access must be implemented consistently across endpoint IP 506 addresses on transports allowing multiple IP addresses per 507 endpoint, for example, SCTP. In particular, the Steering Tag must 508 be consistently scoped and must address the same buffer across all 509 IP address associations belonging to the endpoint. Additionally, 510 operation ordering relationships across IP addresses within an 511 association (set(), get(), etc.) depend on the underlying 512 transport. If the above consistency relationships cannot be 513 maintained by a transport endpoint, then the endpoint is unsuitable 514 for a DDP connection. 516 Multidestination data delivery is a transport characteristic which 517 may require specific consideration in a DDP protocol. As mentioned 518 above, the basic DDP model assumes that buffer address values 519 returned by ddp_register() are opaque to the client protocol, and 520 can be implementation dependent. The most natural way to map DDP 521 to a multidestination transport is to require all receivers produce 522 the same buffer address when registering a multidestination 523 destination buffer. Restriction of the DDP model to accommodate 524 multiple destinations involves engineering tradeoffs comparable to 525 those of providing non-DDP multidestination transport capability. 527 A registered buffer is identified within DDP by its stag_t, which 528 in turn is associated with a socket. This registration therefore 529 grants a capability to the DDP peer, and the socket (using the 530 underlying properties of its chosen transport and possible 531 security) identifies the peer and authenticates the stag_t. 533 The same buffer may be enabled by ddp_post_recv() on multiple 534 sockets. In this case any ddp_recv() untagged message reception 535 indication may be provided on a different socket from that on which 536 the buffer was posted. Such indications are not ordered among 537 multiple DDP sockets. 539 When multiple sockets reference an untagged message reception 540 buffer, local interfaces are responsible for managing the 541 mechanisms of allocating posted buffers to received untagged 542 messages, the handling of received untagged messages when no buffer 543 is available, and of resource management among multiple sockets. 544 Where underprovisioning of buffers on multiple sockets is allowed, 545 mechanisms should be provided to manage buffer consumption on a 546 per-socket or group of related sockets basis. 548 Architecturally, therefore, DDP is a flexible and general paradigm 549 which may be applied to any variety of transports. Implementations 550 of DDP may, however, adapt themselves to these differences in ways 551 appropriate to each transport. In all cases the layering of DDP 552 must continue to express the transport's underlying 553 characteristics. 555 2.2. Remote Direct Memory Access (RDMA) Protocol Architecture 557 Remote Direct Memory Access (RDMA) extends the capabilities of DDP 558 with two primary functions. 560 First, it adds the ability to read from buffers registered to a 561 socket (RDMA Read). This allows a client protocol to perform 562 arbitrary, bidirectional data movement without involving the remote 563 client. When RDMA is implemented in hardware, arbitrary data 564 movement can be performed without involving the remote host CPU at 565 all. 567 In addition, RDMA specifies a transport-independent untagged 568 message service (Send) with characteristics which are both very 569 efficient to implement in hardware, and convenient for client 570 protocols. 572 The RDMA architecture is patterned after the traditional model for 573 device programming, where the client requests an operation using 574 Send-like actions (programmed I/O), the server performs the 575 necessary data transfers for the operation (DMA reads and writes), 576 and notifies the client of completion. The programmed I/O+DMA 577 model efficiently supports a high degree of concurrency and 578 flexibility for both the client and server, even when operations 579 have a wide range of intrinsic latencies. 581 RDMA is layered as a client protocol on top of DDP: 583 Client Protocol 584 | ^ 585 Sends | | Send reception indications 586 RDMA Read Requests | | RDMA Read Completion indications 587 RDMA Writes | | RDMA Write Completion indications 588 v | 589 RDMA 590 | ^ 591 untagged messages | | untagged message delivery 592 tagged messages | | tagged message delivery 593 v | 594 DDP+---> data placement 595 ^ 596 | transport messages 597 v 598 . . . 600 In addition to in-line data flow, read (get()) and update (set()) 601 operations are performed on buffers registered with RDMA as a 602 result of RDMA Read Requests and RDMA Writes, respectively. 604 An RDMA `buffer' extends a DDP buffer with a get() operation that 605 retrieves the value of the octet at address `a': 607 typedef struct { 608 const address_t start; 609 const address_t end; 610 void set(address_t a, data_t v); 611 data_t get(address_t a); 612 } rdma_buffer_t; 614 2.2.1. RDMA Operations 616 The RDMA layer provides: 618 void rdma_send(socket_t s, message_t m); 619 void rdma_write(socket_t s, message_t m, ddp_addr_t d, 620 rdma_notify_t n); 621 void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d); 622 void rdma_post_recv(socket_t s, bdesc_t b); 623 rdma_ind_t rdma_recv(socket_t s); 624 bdesc_t rdma_register(socket_t s, rdma_buffer_t b, 625 bmode_t mode); 626 void rdma_deregister(bhand_t bh); 627 msizes_t rdma_max_msizes(socket_t s); 629 Although, for clarity, these data transfer interfaces are 630 synchronous, rdma_read() and possibly rdma_send() (in the presence 631 of Send flow control), can require an arbitrary amount of time to 632 complete. To express the full concurrency and interleaving of RDMA 633 data transfer, these interfaces should also be reentrant. For 634 example, a client protocol may perform an rdma_send(), while an 635 rdma_read() operation is in progress. 637 rdma_notify_t 639 RDMA Write notification information, used to signal that the 640 message represents the final fragment of a multi-segmented 641 RDMA message: 643 typedef struct { 644 boolean_t notify; 645 rdma_write_id_t i; 646 } rdma_notify_t; 648 identical in function to ddp_notify_t, except that the type 649 rdma_write_id_t may not be equivalent to ddp_msg_id_t. 651 rdma_write_id_t (scalar) 652 an RDMA Write identifier. 654 rdma_ind_t 656 a Send message, or an RDMA error: 658 typedef union { 659 recv_message_t m; 660 rdma_err_t e; 661 } rdma_ind_t; 663 rdma_err_t 665 an RDMA protocol error indication. RDMA errors include buffer 666 addressing errors corresponding to ddp_err_ts, and buffer 667 protection violations (e.g. RDMA Writing a buffer only 668 registered for reading). 670 bmode_t 672 buffer registration mode (permissions). Any combination of 673 permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE) 674 operations. 676 rdma_send(socket_t s, message_t m) 678 send a message, delivering it to the next untagged RDMA buffer 679 at the remote peer. 681 rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n) 683 RDMA Write to remote buffer address d. 685 rdma_read(socket_t s, ddp_addr_t s, length_t l, ddp_addr_t d) 687 RDMA Read l octets from remote buffer address s to local 688 buffer address d. 690 rdma_post_recv(socket_t s, bdesc_t b) 692 post a registered buffer to accept a single Send message, to 693 be filled and returned in-order to a subsequent caller of 694 rdma_recv(). As with DDP, buffers may be enabled on multiple 695 sockets, in which case ordering guarantees are relaxed. Also 696 as with DDP, local interfaces must manage the mechanisms of 697 allocation and management of buffers posted to multiple 698 sockets. 700 rdma_recv(socket_t s); 702 get the next received Send message, RDMA Write completion 703 identifier, or RDMA error. 705 rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode) 707 register a buffer for RDMA on a socket (for read access, write 708 access or both). As with DDP, the same buffer may be 709 registered multiple times on the same or different sockets, 710 and different buffers may refer to portions of the same 711 underlying addressable object. 713 rdma_deregister(bhand_t bh) 715 remove a registration from a buffer. 717 rdma_max_msizes(socket_t s) 719 get the current maximum Send (max_untagged) and RDMA Read or 720 Write (max_tagged) operations that will fit in a single 721 transport message. The values returned by rdma_max_msizes() 722 are closely related to the values returned by 723 ddp_max_msizes(), but may not be equal. 725 2.2.2. Transport Characteristics In RDMA 727 As with DDP, RDMA can be used on transports with a variety of 728 different characteristics that manifest themselves directly in the 729 service provided by RDMA. Also as with DDP, the fundamental 730 characteristics of the transport will not be changed by the 731 presence of RDMA. 733 Like DDP, an RDMA protocol must specify how: 735 o set()s, 737 o get()s, 739 o Send messages, and 741 o RDMA Read completions 743 are ordered among themselves and how they relate to corresponding 744 operations on the remote peer(s). These relationships are likely 745 to be a function of the underlying transport characteristics. 747 There are some additional characteristics of RDMA which may 748 translate poorly to unreliable or multipoint transports due to 749 attendant complexities in managing endpoint state: 751 o Send flow control 753 o RDMA Read 755 These difficulties can be overcome by placing restrictions on the 756 service provided by RDMA. However, many RDMA clients, especially 757 those that separate data transfer and application logic concerns, 758 are likely to depend upon capabilities only provided by RDMA on a 759 point-to-point, reliable transport. In other words, many potential 760 Upper Layers which might avail themselves of RDMA services are 761 naturally already biased toward these transport classes. 763 3. Security Considerations 765 Fundamentally, the DDP and RDMA protocols themselves should not 766 introduce additional vulnerabilities. They are intermediate 767 protocols and so should not perform or require functions such as 768 authorization, which are the domain of Upper Layers. However, the 769 DDP and RDMA protocols should allow mapping by strict Upper Layers 770 which are not permissive of new vulnerabilities -- DDP and RDMAP 771 implementations should be prohibited from `cutting corners' that 772 create new vulnerabilities. Implementations must ensure that only 773 `supplied' resources (i.e. buffers) can be manipulated by DDP or 774 RDMAP messages. 776 System integrity must be maintained in any RDMA solution. 777 Mechanisms must be specified to prevent RDMA or DDP operations from 778 impairing system integrity. For example, threats can include 779 potential buffer reuse or buffer overflow, and are not merely a 780 security issue. Even trusted peers must not be allowed to damage 781 local integrity. Any DDP and RDMA protocol must address the issue 782 of giving end-systems and applications the capabilities to offer 783 protection from such compromises. 785 Because a Steering Tag exports access to a buffer, one critical 786 aspect of security is the scope of this access. It must be 787 possible to individually control specific attributes of the access 788 provided by a Steering Tag on the endpoint (socket) on which it was 789 registered, including remote read access, remote write access, and 790 others that might be identified. DDP and RDMA specifications must 791 provide both implementation requirements relevant to this issue, 792 and guidelines to assist implementors in making the appropriate 793 design decisions. 795 For example, it must not be possible for DDP to enable evasion of 796 buffer consistency checks at the recipient. The DDP and RDMA 797 specifications must allow the recipient to rely on its consistent 798 buffer contents by explicitly controlling peer access to buffer 799 regions at appropriate times. 801 The use of DDP and RDMA on a transport connection may interact with 802 any security mechanism, and vice-versa. For example, if the 803 security mechanism is implemented above the transport layer, the 804 DDP and RDMA headers may not be protected. Such a layering may 805 therefore be inappropriate, depending on requirements. 807 3.1. Security Services 809 The following end-to-end security services protect DDP and RDMAP 810 operation streams: 812 o Authentication of the data source, to protect against peer 813 impersonation, stream hijacking, and man-in-the-middle attacks 814 exploiting capabilities offered by the RDMA implementation. 816 Peer connections which do not pass authentication and 817 authorization checks must not be permitted to begin processing 818 in RDMA mode with an inappropriate endpoint. Once associated, 819 peer accesses to buffer regions must be authenticated and made 820 subject to authorization checks in the context of the 821 association and endpoint (socket) on which they are to be 822 performed, prior to any transfer operation or data being 823 accessed. The RDMA protocols must ensure that these region 824 protections be under strict application control. 826 o Integrity, to protect against modification of the control 827 content and buffer content. 829 While integrity is of concern to any transport, it is 830 important for the DDP and RDMAP protocols that the RDMA 831 control information carried in each operation be protected, in 832 order to direct the payloads appropriately. 834 o Sequencing, to protect against replay attacks (a special case 835 of the above modifications). 837 o Confidentiality, to protect the stream from eavesdropping. 839 IPsec, operating to secure the connection on a packet-by-packet 840 basis, is a natural fit to securing RDMA placement, which operates 841 in conjunction with transport. Because RDMA enables an 842 implementation to avoid buffering, it is preferable to perform all 843 applicable security protection prior to processing of each segment 844 by the transport and RDMA layers. Such a layering enables the most 845 efficient secure RDMA implementation. 847 The TLS record protocol, on the other hand, is layered on top of 848 reliable transports and cannot provide such security assurance 849 until an entire record is available, which may require the 850 buffering and/or assembly of several distinct messages prior to TLS 851 processing. This defers RDMA processing and introduces overheads 852 that RDMA is designed to avoid. In addition, TLS length 853 restrictions on records themselves impose additional buffering and 854 processing, for long operations which must span multiple records. 855 TLS therefore is viewed as potentially a less natural fit for 856 protecting the RDMA protocols. 858 Any DDP and RDMAP specification must provide the means to satisfy 859 the above security service requirements. 861 IPsec is sufficient to provide the required security services to 862 the DDP and RDMAP protocols, while enabling efficient 863 implementations. 865 3.2. Error Considerations 867 Resource issues leading to denial-of-service attacks, overwrites 868 and other concurrent operations, the ordering of completions as 869 required by the RDMA protocol, and the granularity of transfer are 870 all within the required scope of any security analysis of RDMA and 871 DDP. 873 The RDMA operations require checking of what is essentially user 874 information, explicitly including addressing information and 875 operation type (read or write), and implicitly including protection 876 and attributes. The semantics associated with each class of error 877 resulting from possible failure of such checks must be clearly 878 defined, and the expected action to be taken by the protocols in 879 each case must be specified. 881 In some cases, this will result in a catastrophic error on the RDMA 882 association, however in others, a local or remote error may be 883 signalled. Certain of these errors may require consideration of 884 abstract local semantics. The result of the error on the RDMA 885 association must be carefully specified so as to provide useful 886 behavior, while not constraining the implementation. 888 4. IANA Considerations 890 IANA considerations are not addressed in by this document. Any 891 IANA considerations resulting from the use of DDP or RDMA must be 892 addressed in the relevant standards. 894 5. Acknowledgements 896 The authors wish to acknowledge the valuable contributions of 897 Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow. 899 6. Informative References 901 [DAFS] 902 DAFS Collaborative, "Direct Access File System Specification 903 v1.0", September 2001, available from 904 http://www.dafscollaborative.org 906 [FCVI] 907 ANSI Technical Committee T11, "Fibre Channel Standard Virtual 908 Interface Architecture Mapping", ANSI/NCITS 357-2001, March 909 2001, available from http://www.t11.org/t11/stat.nsf/fcproj 911 [IB] InfiniBand Trade Association, "InfiniBand Architecture 912 Specification Volumes 1 and 2", Release 1.1, November 2002, 913 available from http://www.infinibandta.org/specs 915 [MYR] 916 VMEbus International Trade Association, "Myrinet on VME 917 Protocol Specification", ANSI/VITA 26-1998, August 1998, 918 available from http://www.myri.com/open-specs 920 [ROM] 921 A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP 922 Problem Statement", draft-ietf-rddp-problem-statement, 923 Internet Draft Work in Progress 925 [SCTP] 926 R. Stewart et al., "Stream Transmission Control Protocol", RFC 927 2960, Standards Track 929 [SDP] 930 InfiniBand Trade Association, "Sockets Direct Protocol v1.0", 931 Annex A of InfiniBand Architecture Specification Volume 1, 932 Release 1.1, November 2002, available from 933 http://www.infinibandta.org/specs 935 [SRVNET] 936 R. Horst, "TNet: A reliable system area network", IEEE Micro, 937 pp. 37-45, February 1995 939 [VI] Compaq Computer Corp., Intel Corporation and Microsoft 940 Corporation, "Virtual Interface Architecture Specification 941 Version 1.0", December 1997, available from 942 http://www.vidf.org/info/04standards.html 944 Authors' Addresses 946 Stephen Bailey 947 Sandburst Corporation 948 600 Federal Street 949 Andover, MA 01810 USA 950 USA 952 Phone: +1 978 689 1614 953 Email: steph@sandburst.com 955 Tom Talpey 956 Network Appliance 957 375 Totten Pond Road 958 Waltham, MA 02451 USA 960 Phone: +1 781 768 5329 961 Email: thomas.talpey@netapp.com 963 Full Copyright Statement 965 Copyright (C) The Internet Society (2005). This document is 966 subject to the rights, licenses and restrictions contained in BCP 967 78 and except as set forth therein, the authors retain all their 968 rights. 970 This document and the information contained herein are provided on 971 an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE 972 REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND 973 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, 974 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT 975 THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR 976 ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A 977 PARTICULAR PURPOSE. 979 Intellectual Property 980 The IETF takes no position regarding the validity or scope of any 981 Intellectual Property Rights or other rights that might be claimed 982 to pertain to the implementation or use of the technology described 983 in this document or the extent to which any license under such 984 rights might or might not be available; nor does it represent that 985 it has made any independent effort to identify any such rights. 986 Information on the procedures with respect to rights in RFC 987 documents can be found in BCP 78 and BCP 79. 989 Copies of IPR disclosures made to the IETF Secretariat and any 990 assurances of licenses to be made available, or the result of an 991 attempt made to obtain a general license or permission for the use 992 of such proprietary rights by implementers or users of this 993 specification can be obtained from the IETF on-line IPR repository 994 at http://www.ietf.org/ipr. 996 The IETF invites any interested party to bring to its attention any 997 copyrights, patents or patent applications, or other proprietary 998 rights that may cover technology that may be required to implement 999 this standard. Please address the information to the IETF at ietf- 1000 ipr@ietf.org. 1002 Acknowledgement 1003 Funding for the RFC Editor function is currently provided by the 1004 Internet Society.