2.3.13 IP over InfiniBand plus MIBs BOF (infinib)

Current Meeting Report

Meeting minutes for the 50th IETF Infinib BOF
Minutes by: Michael Krause
Vivek Kashyap

Chairs of the BOF:
Bill Strahm
Daniel Cassiday

Opening Remarks
- - Agenda Bashing
- No one complained

- - Overview presentation (Dan Cassiday)

Question: Will IB flow over the Internet?
Answer: IB not designed for Internet and it is unclear if this will be practical. Not the intent of the proposed workgroup tackle this problem.

Question: Is it possible to design a scheme to establish a path without using the SM?
Answer: IP management does not need to use the same mechanism / type of management as IB management, i.e. the centralized approach.

Question: Is an Ethernet NIC a router, a bridge, what?
Answer: Various mechs - direct IB to NIC, router, etc.

Question: Is multicast part of the charter?
Answer: Yes. Later presentation.

Question: What is used to composed a link layer address?
Answer: Different techniques. Dealt with in next presentation.

- - IP over IB Address Resolution (Vivek Kashyap)
Question: How are different PMTU handled?
Answer: This is an open issue to be resolved by the workgroup. Impacted by transport type, bridging layer, etc.

Question: SAR would be provided by IB transport?
Answer: Yes.

Question: Numerous questions on address resolution and the impact of centralized versus distributed.
Answer: Numerous answers but basically left as part of workgroup to decide.

- - IP over IB Encapsulation (Jerry Chu)
Question: Why not have only one MTU?
Answer: Higher performance implementation by using hardware for SAR. Requires connected IB transports to be used. Single MTU applies toUD transport type (IB does not SAR UD operations).

Good commentary on the complexity of IB - what's a little more when it comes to multicast?

- - IP Subnets / Partitioning / QoS (Mike Krause)

Question: Clarifying questions about addressing, SL, etc.

- - Performance Management (Shravan Pargal)

Question: Minor clarification of 32 vs 64-bit counters / set / reset
Answer: Native IB counters are 32 bit. However it would be possible in software to handle the required additional "state" to produce the required 64 bit counters.
- - Storage Aspects of InfiniBand (Roger Cummings)

Question: Can we get to a single SCSI MIB for all transports?
Answer: T10, IPS, etc. working on potential standards for SCSI with separate for iSCSI. There would be something for IB as well. Can all of this be combined? Attend the interim meetings and provide input.

Q: The GRH equivalence to IPv6 can be used for IB communication ?
A: IB is designed for data room interconnect and does not hold up well for longer distance communication. Additionally this comes under the ambit of IB over IP which is a different subject.

Q: What of ethernet connectivity ?
A: There are multiple options. The CA (with the ethernet NIC) can be a bridge, router, or the NIC can be addressed directly over IB.

Q: Can SM be avoided in path determination ?
A: How address resolution is done need not depend on the subnet manager.

Q: Is multicast supported ?
A: Yes and an IP encapsulation will be defined

Q: Is there a unique physical address associated with end points
A: Yes, an EUI-64 identifier is associated with each port and with each channel adapter/switch/router.

Q: What is a Link Address ?
A: Presented in detail in 'IPoIB address resolution' by Vivek Kashyap

Q: Why a link address with multiple identifiers ? One can be used and then the other information recovered.
A: That is a possible solution. However, there are pros and cons to using any of the port identifiers. The various options are described here and determining the best method will be the task of the WG.

Q: Does IB define a path MTU discovery method ?
A: No such method is defined. The IB path MTU is determined during connection setup or can be got from the Subnet Manager/Administrator.

Comment: A multicast cloud need not be defined. A point-to-point model can be used.

Q: How common will multicast support be in IB fabrics ?
A: Most switch vendors are already supporting or are planning to support multicast.

Comment: There is a scalability issue with multicast since it is centered at SM.
This must be mitigated by the right solution.

Q: What are the MTUs being supported by vendors
A: Intel is shipping with 256 bytes at present but intends to support 2K in the next release. Most others are implementing larger MTUs.

Q: IB uses addresses similar to the IPv6 addresses. Is there an IPv6 equivalent link local address ?
A: Yes, however, it is limited to an IB subnet.

Q: GRH being same as IPv6 allows use of direct IPv6 packets
A: Yes, but IB's Raw IPv6 datagram mode uses only 16 bit CRC (and not 32 bit) does not support QPs and partitioning.

Q: Are IB router protocols and multicast across IB subnets defined ?
A: No, not yet but are being defined.

Q: Is IBTA looking towards IETF for help ?
A: IBTA wants to leverage IETF's experience and use that to determine the solution for IBTA only solutions too. So, IPoIB issues will have an impact on the evolving IB specification in this area.

Q: Is the SL to VL mapping and the corresponding arbitration for flow control defined ?
A: Yes, an interoperable standard exists and is fully defined such that any compliant switch will interoperate with other compliant nodes.

Q: Why not communicate using a regular LAN ?
A: The servers determined for IB will reside in a rack with IB only connections. The LAN connection will be in an I/O chassis that is in another rack in the data center. It makes sense for the servers that are situated a few inches away from one another to talk directly rather than going across the IB interconnect to LAN and then back again. Secondly, every IB node will have to support a LAN interconnect.

Comment: The MIB counters are not represented as presented in the
performance counter slides but this can be sorted out later.

Comment: Define a single SCSI MIB for all transports Discussion on various other bodies and forums that are looking into this t10, ips etc.

Wrap up:

Thomas Narten enquired on the number of people interested in participating in the WG if formed by a show of hands. The ADs (and other relevant parties) will decide on forming the WG.

The draft charter looks good and will be discussed further as needed.


One strong objection from Steve Deering (Cisco). Steve's objection was that the IETF was getting to thin on IP over foo issues and people should just quit inventing layer 2 protocols.

After debate the objection was withdrawn with the conclusion that people weren't going to quit inventing layer 2 protocols, and it would be unwise not to give Layer 2 developers the experience of the IETF to run IP over these protocols.

Charter will be discussed on IPoverIB reflector next week.



IP over InfiniBand: Subnets / Partitions / QoS
Storage Aspects of IB
IPoIB Address Resolution
InfiniBandtm Overview & Charter
IP over IB Encapsulation Unicast and Multicast