Current Meeting Report

2.7.18 RDMA over the Internet Protocol Suite (roi) Bof

Current Meeting Report

Minutes from the RDMA over IP BOF, IETF 53
Tuesday, March 19, 2002, 1-2PM

Chaired by Allyn Romanow and Steph Bailey.
Scribed by Jeff Hilland

Agenda, Intro - Steph Bailey
The Problem - Tom Talpey - 15 min
Scenario, Storage - David Black - 20 min
Charter - Allyn Romanow - 20 min

( Quick Headcount - room seated 540 with about 130 in attendance. )

Agenda Bashing. Steph asked how many people had read the problem statement. Looked to be about half of the group.

Ground Rules were covered - Focus on the Problem and if this work should be done in the IETF and the proposed charter. Implementation and Architecture are out of scope for the BOF.

Tom Talpey then covered the problem statement, as one of the authors. This covered the problem, context, terminology, high-level review. The problem as stated in the document is the I/O overhead and memory bottleneck. Memory and network are in parity right now, but with 10Gb there is a 10x disparity. The current memory access model at the receiver is a 3-touch model. The receiver is the problem because here data copy is unavoidable in the general case. Some zero copy mechanisms exist but they do not solidly fix the problem.

He also put forth the proposition that IP is under-used as a high-speed interconnect due to the overhead of the IP protocol suite. He mentioned that this is targeted at Server class machines for storage, clusters, databases which need high performance & low overhead communications. The networks here are geographically constrained, low loss, med/high bandwidth and low latency.

He then stated the benefits as the services spread outward over public networks for remote datacenters or clients that see benefits of this technology. The point was made that the right solution is desired at it should be general enough and flexible enough to support all of these needs.

The concept required would be something that does zero copy. This is decomposable into two distinct parts: DDP and RDMA. Direct Data Placement requires information to be carried with the packet to indicate the final destination in memory. RDMA contains the semantics allowing access by trusted peers to selected local memory using DDP techniques. He then provided a high level example of RDMA Write. He concluded with the relevance of the problem in the industry today and how early IETF guidance and influence will avoid later problems. Concerns include harmony with the IP protocol suite and security - issues the IETF has great expertise in.

Sally Floyd, IAB, asked why the IETF? Were the other solutions not successful? Tom answered this question to point out that the other solutions are successful in limited fashion. This body of work should be done in the IETF because IETF controls the IP transports.

Another point was raised that separation [what separation?] isn't very clean in reality. An excellent point was raised that this needs to be done in the IETF because the others are all proprietary and thus not interoperable. John Wroclawski reiterated that RDMA may not be the only solution and others should not be taken off of the list a priori.

David Black then proceeded into the iSCSI over SCTP scenario. He described the 3 buffer copies in iSCSI and what happens when packets are dropped and how that can result in worse problems. This results in needing 3-5 Gb of memory bandwidth on a 1Gb Enet iSCSI link. At 10Gb this becomes unacceptable. So what can be done today? Protocol specific logic in data receive path is one solution, but when you have to do it for each protocol this means more receive path code, and if a chunk is dropped it all grinds to a halt. So can we do better? He delineated the characteristics of a solution: fast, extensible and matches DDP functionality and works great for existing protocols without changing flow control or major protocol parts.

How can RDMA help? Buffer management is a necessary evil and DDP does not help this but RDMA can help here. He then answered the question of why the IETF? Key motivation was use with multiple protocols, getting the security right and transport interaction right as well as leveraging the knowledge of existing protocols (iSCSI, NFS). In summary the goal is protocol independent data copy avoidance through DDP functionality and RDMA semantics.

There was a question about buffer management is a pain because the other end is doing weird stuff and you can't control it. There was a misunderstanding of who is managing the buffers. The issue is by exporting those addresses you are forcing conformance to your management scheme. This is only partially true. So what is the cost penalty? Is it higher than the buffer copy? David's answer is that the ULP should negotiate it. Another clarification was then brought up about the addressing. Then a comment came up that pushing the memory management overhead from server to client is not true. But a hidden cost may be that memory registration/deregistration must be done on every transaction do be secure. David said this is outside of the agenda and that it was mischaracterized. A possibility was brought up that DDP may be able to be done at the network layer in IPv6 that should not be taken off the table a priori. A comment was made on buffer management - RDMA exports the management of the names of the buffers - a handle. The other side doesn't have to understand space management. This is a distributed protocol with questions about the lifetime about the descriptor of the buffer. Time was called with 7 queued at the mike.

Allyn then covered the Proposed Charter. Workgroup items were the problem statement, terminology document, transport independent protocol design including layering and security and mapping of the protocol design onto SCTP. She then put some milestones up - Aug 02: Submit problem statement to IESG and submit terminology draft to IESG as informational RFCs. Jan03 submit core protocol specs and submit the SCTP mapping to IESG for standards track pubs.

Questions: There are alternative solutions and that may be missing from the charter. Document that is missing is the one that discusses the alternatives. Allyn thought this is a great idea but requires people come forward with alternatives. Another comment was that TCP was conspicuously missing. Allyn answered that this was per the area directors. Another comment came up the same. Allison stated that SCTP is a first class protocol in the transport area. When you use TCP, you are putting the data into memory when you receive it. Another comment was that requirements for the transport protocol are missing. (editor's note: these requirements were in the last set of documents). Another comment was that multi-homing isn't mentioned and needs to be covered. The mike's were then cut off for the ADs.

Allison took a humm for people who think that this is important to be in the IETF. Lots of Humm. Those against - dead silence.

Based on the charter that has been presented, with some modifications - is this ready to be chartered to be a working group. Lots of Humm for. Those against - 2 or 3 people humming (that I could hear).


ROI: A Storage Scenario
Problem Statement "RDMA Over Internet Protocol Suite"