2.8.13 Remote Direct Data Placement (rddp)

Last Modified: 2003-07-28

David Black <black_david@emc.com>
Transport Area Director(s):
Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>
Transport Area Advisor:
Jon Peterson <jon.peterson@neustar.biz>
Technical Advisor(s):
Allyn Romanow <allyn@cisco.com>
Stephen Bailey <steph@cs.uchicago.edu>
Mailing Lists:
General Discussion: rddp@ietf.org
To Subscribe: rddp-request@ietf.org
In Body: subscribe
Archive: www.ietf.org/mail-archive/working-groups/rddp/current/maillist.html
Description of Working Group:
The purpose of this WG is to enable Remote Direct Data Placement (rddp)
capabilities with IP transport protocols, in particular with
SCTP. RDDP capabilities refer to the ability to place data directly
from the NIC into application buffers, without intensive CPU
usage. This strategy avoids the costs of data copying and enables
using IP as a method for high speed buffer to buffer transfer,
allowing IP to replace special purpose networks currently in use.
Remote Direct Memory Access (RDMA) is an example of this concept.

Conceptually, RDDP functionality can be viewed as consisting of two
layers. First the direct data placement capability, which is
accomplished through a tag and a lookup table on the NIC. Above this
core functionality, an RDDP control protocol is needed to specify how
the direct data placement can be used, for example READ and WRITE

The work of the WG is to accomplish four items:

1) A (transport independent) protocol core to support direct data
  placement from the NIC into specified memory, usually application

2) A (transport independent) protocol core layered on top of the direct
  data placement protocol that specifies control of RDDP.

3) A mapping of the direct data placement protocol onto SCTP,
  for standards track, including a clear applicability statement
  of the expected service from the mapping.

4) A mapping of the direct data placement protocol onto TCP, for
  informational, because TCP's service is a less good match to RDDP,
  including an applicability statement of the issues regarding the
  service available from the mapping.

The working group will ensure that the resulting technology will be
secure and will not enable new attacks on systems supporting RDDP.

The WG will not modify existing Internet transport protocols, but
may forward issues it discovers in such transport protocols that
are not full Internet Standards to the appropriate IETF WGs for
their consideration.
Goals and Milestones:
Done  Submit Internet-Draft including problem statement and architecture
Done  Submit Initial draft of Remote Direct Data Placement protocol (RDDP)
Done  Submit Initial draft of RDMA control protocol, to be named.
Done  Initial draft mapping the RDDP core and control onto SCTP including A/S
Jun 03  Submit problem statement and architecture drafts to IESG for consideration as informational publications
Done  Initial draft of security considerations for RDDP
Done  Initial draft of informational mapping of the RDDP core and control onto TCP
Done  Initial draft of applicability statement covering both the SCTP and TCP mappings
Dec 03  Submit RDMA control protocol (named TBD) to IESG for consideration as a Proposed Standard
Dec 03  Submit Remote Data Placement Protocol (RDDP) to IESG for consideration as a Proposed Standard
Dec 03  Submit RDDP security considerations draft to IESG
Mar 04  Consult Area Directors about any additional work the WG should undertake
Mar 04  Submit to IESG: - mapping of the RDDP core and RDMA control protocols onto SCTP for consideration as proposed standard; - mapping of the RDDP core and RDMA control protocols onto TCP for consideration as informational publications; - applicability statement for RDDP core and RDMA control protocols over both SCTP and TCP for consideration as an informational publication
  • - draft-culley-iwarp-mpa-03.txt
  • - draft-stewart-rddp-sctp-02.txt
  • - draft-ietf-rddp-rdma-concerns-00.txt
  • - draft-ietf-rddp-arch-02.txt
  • - draft-ietf-rddp-problem-statement-02.txt
  • - draft-ietf-rddp-rdmap-00.txt
  • - draft-ietf-rddp-ddp-00.txt
  • - draft-ietf-rddp-applicability-00.txt
  • - draft-pinkerton-rddp-security-00.txt
  • No Request For Comments

    Current Meeting Report

    RDDP WG minutes - Vienna, Austria, July 2003 
    Thanks to Tom Talpey for taking the initial minutes. Letters in [square 
    brackets] indicate first letter in name of corresponding 
    presentation in proceedings. 
    Tuesday, July 15 5-6pm 
    David Black (WG chair) introduces agenda changes, no bashing, agenda 
    accepted [A] 
    Review of WG draft intentions and status - see presentation [B] 
    + RDDP Applicability statement, Lode Coene [C] 
    Review of document content - draft considered by authors to be in good 
    shape, waiting for comments from WG. 
    David Black requests that TCP mapping issues (drawbacks of TCP by 
    comparison to SCTP) be included in the applicability statement. 
    + SCTP Mapping draft [D] 
    On its third revision. 
    A DDP Stream establishment issue has been identified, can a 
    protection domain assignment be deferred or no? Needs 
    documentation at a minimum. 
    A draft revision under author review concerns single-message exchange for 
    "private data", compatible with current MPA draft, but this is part of the 
    open startup issues in the MPA draft. 
    No others issues are currently identified, speak up now... New version 
    expected in next few weeks. 
    + Shared receive queues vs Shared buffer pool [D] 
    David Black presented this issue from the mailing list on behalf of 
    Caitlin Bestler, who was unable to travel to Vienna. 
    Shared receive queues are proposed in 
    draft-hilland-verbs-00. Their presence/absence has impact on other 
     Architecture document should it define a "post receive"? Security issues 
    The problem: need to under provision receive buffers, but how to control 
    per-receive queue quotas? 
    Two proposals: Shared receive queue (draft-hilland) Shared buffer pool 
    (email on list) 
    The proposals are described and a lengthy discussion ensues. 
    David Black suggests trying the following tack: need to resolve 
    robustness to DOS attack on shared buffers. Tag this as the only open 
    portion of this issue and look at countermeasures in security draft. 
    Examine offline and continue this issue on Thursday. 
    A question is asked about why this issue is holding up the 
    architecture draft, which is *not* an interface specification. David Black 
    answers that the architecture draft goes into sufficient detail on 
    pseudo-interfaces that some discussion of resource sharing is 
    necessary there. 
    + RDMAP/DDP Security draft - Jim Pinkerton [E] 
    Presents the new security draft for first time. 
    Draft is focused on wire protocol issues, in a generic way with out 
    referencing verbs, etc. 
    Sketches components necessary to implement security over RDMA. - 
    privileged resource manager, controlling the RNIC itself as well as two 
    classes of application - priv and non-priv applications. 
    Identifies resources to be analyzed (and controlled): connection 
    context, data buffers, translation tables, STags, completion queues, plus 
    RDMA Read request queue. 
    Describes the "dimensions of trust": local resource sharing, local trust and 
    remote trust. 
    There are 8 combinations of trust, 4 are addressed in the paper. The 4 were 
    chosen as most applicable to IETF areas of interest, due to brevity and 
    also lack of relevant applications which might fall into the other 4. 
    Enumerates tools for countermeasures: protection domain, limiting STag 
    scope, (4 others) 
    <presentation abbreviated due to end of WG meeting time> 
    David Black requests that for Thursday's session, protocols should decide 
    which "trust bucket" each lands in, determines if bucket is addressed in 
    draft, and if so are the issues accurate? 
    Thursday, July 17, 3:30-5:30pm 
    + Architecture draft resolution 
    ddp_post_receive goes back in Allow sharing of buffers for untagged 
    receives Work out how to avoid tacking limit per stream/socket problem in 
    arch draft; problem will be addressed in security draft In ddp draft, ok to 
    not fail immediately on shared overdraw 
    question from floor - have we addressed the issues raised by VJ 
    delay-bandwidth results? Enough buffers to allow the flow to use all 
    available bw? A (David Black): The fact that RDDP moves control of 
    buffering resources upwards in the protocol stack doesn't change the need to 
    provide bandwidth-delay product quantity of buffering to enable full use of 
    the available network bandwidth. This is an issue for receivers to 
    appropriately provision receive buffers. Consider adding a statement to 
    this effect to the architecture draft. 
    + Return to Security document discussion, Jim Pinkerton. 
    [This meeting time conflicts with SAAG meeting so security experts may not be 
    in attendance.] 
    Current open issues summarized: - IPSec section is currently a 
    placeholder, and will be filled in (Bernard Aboba to help out) - Also, 
    summary section at end still in progress. - From reflector, elaborate some 
    issues in "trust model". - Multiple Protection Domains - Scope of STags - 
    Other resources to add to security discussion (async event queue, PD, etc) - 
    STag disable/enable by non-privileged app. 
    Question from floor - we need an analysis of issues introduced by RDMA into 
    existing protocols, e.g. NFSv4. When transfers are done purely by RDMA they 
    may not be protected by existing security. A (David Black): This may 
    indeed require additional security coverage such as IPsec which may be 
    enabled by existing CCM NFSv4 proposal. 
    It as also noted that the integrity protection is present in NFSv4, 
    RPCSEC_GSS integrity, can verify that RDMA transfers actually occur as 
    promised, while IPsec can be used to enforce the authorizations 
    themselves. Two different issues, important to keep separate. 
    Question - should performance analysis be included in our analysis? 
    Especially where IPSec overhead is involved. A (David Black): First of all, 
    let's complete the security analysis. At that point, the specific 
    performance implications may be worth analyzing. But not before. 
    Next major revision of this document should become an official WG draft. No 
    + Status of TCP mapping issues 
    WG chair summarizes marker issues resolved to date: Sender alignment 
    Marker usage, intervals, optionality CRC use (5+ issues) Further 
    discussion to go into applicability draft 
    Question from floor - have we worked out when it's ok not to use MPA CRC 
    yet? Prefers to have it always present for simplicity. A (David Black): 
    Rough consensus is for CRC to be optional. Discussion of when it may be 
    disabled and what ULP designers/implementers should assume goes into the 
    applicability statement draft. 
    Last(?) CRC Issue - one CRC or two? Case for two - faster placement, fast 
    header validation Case for one - tractable, simpler, doesn't abuse 
    Statement from floor - there are more reasons to have two CRCs: DDP 
    Routers don't have to rebuild data crc Don't have to wait for whole 
    segment to begin processing header 
    Large amount of controversy whether it's valid to rewrite DDP headers. 
    WG Chair declares that rough consensus exists for one CRC; while there are 
    advantages to two, they don't outweigh increased complexity. Further 
    discussion will be on the list. 
    + MPA discussion - Paul Culley 
    Changes to most recent draft recapped: optional, 
    receiver-specified markers optional, mutually agreed CRC MPA streaming mode 
    "key" validation mechanism at connect new informational sections 
    Authors' additional proposals no "alignment" of market to TCP sequence #, as 
    alignment would violate layering CRC field always present, not filed in or 
    checked when not used - simplicity no private data in MPA (carried only in 
    TCP mode), quiescing stream required at MPA init MPA 128-bit start key, 
    carrying a fixed MPA signal + CRC/marker bits, etc. 
    passive/active rules specified Question - what happens if both 
    "active"? Needs to be addressed Resolved - Start Key should at a minimum 
    assert Active vs. Passive. 
    Discussion of connection startup models - immediate RDMA enabled, RDMA 
    enabled midstream, RDMA midstream with private data (blob follows MPA 
    key). This whole area of how to initiate a connection, including private 
    data, is open and needs to go to list 
    Chair proposes that this document is ready to go to official WG draft for 
    TCP mapping of RDDP (with significant issues to be worked out of 
    course). No objections. 
    Question: Is the MPA draft as informational or standards track? A (WG 
    Chair) Informational: per ADs from recent rechartering. Can be 
    reconsidered when draft is complete and ready for publication as an RFC. 
    Q: Didn't acceptance of Start Key mechanism enable standards track 
    status? A (WG Chair): No, refusal to accept Start Key mechanism would have 
    resulted in TCP mapping work being removed from RDDP WG charter. 
    NOTE: WG Chair has subsequently verified w/ADs that standards track 
    status for the TCP mapping (MPA) can be reconsidered when the draft is 
    complete. Among the reasons for this is a strong desire to see exactly what 
    is being proposed and understand its effects on TCP before approving 
    standards track status. 


    IETF RDDP WG Draft Status
    RDDP Applicability Statement
    SCTP Draft
    RDMAP/DDP Security Draft
    RDDP TCP Mapping: Rough Consensus Summary
    MPA MPA update (-03)