RDMA/IP Mini-BOF - minutes STORM (STORage Maintenance) WG, IETF-88 Vancouver Canada 1300-1500 Thursday Nov 7, Regency B Co-Chairs: David Black, Tom Talpey Slides are here (scroll down to storm WG): https://datatracker.ietf.org/meeting/88/materials.html#tsv Blue Sheets, Introduction and Agenda Bashing - David Black - 5 min - Done. RDMA Ecosystem - Tom Talpey - 20 min What is RDMA Previous IETF work Upper layers using RDMA iSER, NFS/RDMA, SMB3, SCSI RDMA Protocol (SRP), High Performance/Scientific/Financial Computing, Experimental RDMA requirements of each RDMA layers iWARP, InfiniBand, RDMA over Converged Ethernet (RoCE) Lower layers used by RDMA TCP, SCTP, Ethernet L2, Infiniband RDMA trends and development Virtualization, encapsulation Fabrics, infrastructure, bandwidths -> Overview presentation, see slides -> Q&A: Only SRIOV shown on slides for device virtualization - any form of IOV is fine, e.g., MRIOV, but SRIOV is more widely deployed. iWARP trends - Brian Hausauer - 15 min+discussion Protocol status Extensions status Work in Progress -> Tom Talpey presented for Brian Hausauer, who had a family emergency, see slides. RoCE trends - Diego Crupnicoff - 15 min+discussion Protocol status Related standards org(s) Work in Progress -> Presentation is primarily on behalf of the IBTA (InfiniBand Trade Association) -> IBTA specifies RoCE (RDMA over Converged Ethernet), and is working on specifying -> Routable RoCE (routable implies that IP is used) see slides. Fabric trends - Pat Thaler - 10 min+discussion RoCE dependencies from (Ethernet) fabric Data Center Bridging (DCB) including Priority Flow Control (PFC) -> Pat is also vice-chair of IEEE 802 and Data Center Bridging task group chair in IEEE 802.1, some of these slides are based on a DCB tutorial. -> Congestion Notification doesn't work well when there's a layer 3 (IP) router in the traffic path. -> Comment: PFC (Priority Flow Control) is good for a small number of hops (1-2), but actual congestion control is likely to be needed at larger scale, as PFC will have issues at larger number of hops. -> Q (Tom Talpey): How to check that PFC has been configured correctly, especially on multi-vendor networks? - A: Can look at DCBX/LLDP results. Most FCoE interfaces won't send traffic into a link if there's no link flow control (PFC or otherwise). General discussion - 1 hour, also included above Future work within IETF Related work within IETF Relationship to external standards org(s) Protocol alternatives, e.g. iWARP, RoCE Identification of areas to investigate Long discussion, three conclusions: (1) There are opportunities for IBTA/IETF collaboration on topics related to Routable RoCE as the IBTA continues their activity to specify it. The storm WG chairs (David Black, Tom Talpey) will continue to explore collaboration possibilities with Diego Crupnicoff (IBTA) and report back as appropriate. (2) There was a lot of interest in the room in congestion control. IETF congestion control work is best done in other WGs in a general fashion, as opposed to designing a congestion control mechanism specific to RDMA. OTOH, RDMA is a fine source of requirements for congestion control. People interested in congestion control should follow up directly with other IETF WGs that are doing congestion control work, e.g., aqm, rmcat. (3) It makes sense to continue to enhance iWARP to obtain functional parity with InfiniBand - this provides RDMA transport choice at upper layers (commonality at RDMA verbs interface and above). An Internet-Draft that specifies the remaining necessary enhancements to iWARP is expected soon, and will be directed to the storm WG. ----- Additional background: The STORM (STORage Maintenance) working group's responsibilities include maintenance and development of various IETF RDMA specifications, and interest and activity is rising in these. Many RDMA device and RDMA-aware upper layer implementations already use IETF-standard protocols, and others are moving toward using IP networks, by defining new IP-family layering. Use of the iWARP family is already in place for three broadly implemented storage protocols (iSCSI, NFS, SMB), as well as for High Performance Computing. The Infiniband Trade Association, which defines the InfiniBand and RDMA over Converged Ethernet (RoCE) transports, is also addressing these. Additional extensions to the IETF RDMA RFCs are in progress in the STORM working group. A BOF is scheduled for the upcoming meeting to review the status of the various RDMA-related efforts, and the requirements of upper layers. With these at front of mind, a discussion will be held to determine interest and desire to proceed with next steps in the IETF, and coordination with external standards efforts. In addition to determining possible future IETF work, the discussion will explore the role IETF may play in external specifications. Example RDMA-capable Transports: iWARP (RFC5040-5044) and draft-ietf-storm-rdmap-ext RoCE (www.infinibandta.org) InfiniBand (www.infinibandta.org) Example Upper layers using RDMA: NFS/RDMA (RFC5666-5667) iSER (RFC5046) and draft-ietf-storm-iser SMB3 and SMB Direct (MS-SMB2 and MS-SMBD http://msdn.microsoft.com/en-us/library/jj712081.aspx) MPI (http://en.wikipedia.org/wiki/Message_Passing_Interface) Additional implementations/use: iWARP extensions (continued) Routable RoCE Shared memory (draft-fox-tcpm-shared-memory-rdma) Further use by storage upper layers Applicability to new storage technologies e.g. flash, nonvolatile memory, etc.