2.3.7 IP over InfiniBand (ipoib)

NOTE: This charter is a snapshot of the 60th IETF Meeting in San Diego, CA USA. It may now be out-of-date.

Last Modified: 2004-07-01

H.K. Jerry Chu <jerry.chu@sun.com>
Bill Strahm <bill@strahm.net>
Internet Area Director(s):
Thomas Narten <narten@us.ibm.com>
Margaret Wasserman <margaret@thingmagic.com>
Internet Area Advisor:
Margaret Wasserman <margaret@thingmagic.com>
Mailing Lists:
General Discussion: ipoverib@ietf.org
To Subscribe: ipoverib-request@ietf.org
In Body: subscribe ipoverib
Archive: http://www.ietf.org/mail-archive/web/ipoverib/index.html
Description of Working Group:
E-mail archive: http://www.ietf.org/mail-archive/working-groups/ipoverib/current/mailli st.html

InfiniBand is an emerging standard intended as an interconnect for processor and I/O systems and devices (see the Infiniband Trade Association web site at http://www.infinibandta.org for details). IP is one type of traffic (and a very important one) that could use this interconnect. InfiniBand would benefit greatly from a standardized method of handling IP traffic on IB fabrics. It is also important to be able to manage InfiniBand devices in a common way.

This working group has two tasks:

- specify the protocols and encapsulations to transport IPv4/v6 over an InfiniBand fabric.

- specify a set of MIB objects to allow management of the InfiniBand fabric itself.

The initial scope of the WG was limited to the use of the basic IB Unreliable Datagram (UD) transport mode for transporting IP over Infiniband. With that work mostly done, the scope has been extended to develop an optional mechanism for transporting IP over other IB transport modes. In particular, there is a desire to transport IP over one or both of IB's connected modes, which enable the use of a much larger MTU than the IB link MTU size. They also provide improved reliability and performance through the use of link level orderly and reliable delivery, and IB's automatic path migration (APM) feature. However, care must be taken to ensure that use of an IB reliable transport does not unduly interfere with the retransmission and congestion control mechanisms used by higher layers (e.g., TCP and SCTP).

Other more advanced functionalities such as mapping IP QOS into IB-specific capabilities remain out of scope of the WG charter.

Work items

1. Specify standards track procedures for transporting IP over IB. This includes:

- supporting ARP/ND packets in order to map IP addresses into IB link-layer addresses.

- define encapsulations for carrying ARP, IPv4 and IPv6 packets over IB

- Define how to transport IP multicast over IB.

2. Specify a standards track channel adapter MIB that allows management of an InfiniBand channel adapter. There will require that InfiniBand types be added to the ifType defined by IANA

3. Specify a standards track baseboard management MIB that will allow management of specified device properties

4. Specify sample counter MIBs to allow InfiniBand sample counters to be exposed to external SNMP management applications

5. Specify an optional, standards track encapsulation for carrying IPv4 and IPv6 packets over either IB unreliable connections or reliable connections.

Goals and Milestones:
Done  Submit initial Internet-Draft of ARP encapsulation
Done  Submit initial Internet-Draft of Requirements/Overview
Done  Submit initial Internet-Draft of IP V4/V6 Encapsulation
Done  Submit initial Internet-Draft of Infiniband-Like MIB
Done  Submit initial Internet-Draft of Channel Adapter MIB
Done  Submit initial Internet-Draft of Multicast
Done  Submit initial Internet-Draft of Baseboard MIB
Done  Submit initial Internet-Draft of Subnet Mangement MIB
Done  Submit ARP/IP/Multicast encapsulation drafts for IESG Last Call
Mar 02  Submit Infiniband-Like MIB for IESG Last Call
Mar 02  Submit Channel Adapter MIB for IESG Last Call
Nov 02  Submit initial Internet-Draft of Advanced Encapsulation over Connected Transports
Mar 03  Submit initial Internet-Draft of Sample Counter MIB
  • - draft-ietf-ipoib-ibif-mib-06.txt
  • - draft-ietf-ipoib-subnet-mgmt-agent-mib-06.txt
  • - draft-ietf-ipoib-ibmib-tc-mib-05.txt
  • - draft-ietf-ipoib-architecture-04.txt
  • - draft-ietf-ipoib-ip-over-infiniband-06.txt
  • - draft-ietf-ipoib-dhcp-over-infiniband-06.txt
  • - draft-ietf-ipoib-channel-adapter-mib-05.txt
  • - draft-ietf-ipoib-baseboard-mgmt-agent-mib-01.txt
  • - draft-ietf-ipoib-perf-mgmt-agent-mib-02.txt
  • - draft-ietf-ipoib-subnet-manager-mib-00.txt
  • No Request For Comments

    Current Meeting Report

    IPoIB Meeting Minutes

    MIB Stataus:

    Not updated

    There is an Open IB group, to implement IB on 2.6 Linux.
    Bill presents Hal's slides.

    January 2004 there was a IBTA Plugfest to test interopbility. August will be another plugfest. Organized by IBTA andOpenIB is participating that. January/Feb 2005 again. Need to resolve NDA issue with IBTA and IETF. Margeret will work with Bill on that.

    check openib.org for update.

    ARP size is too small for IB. Rebuild kernel to make room for it.
    Simply enlarge that will that break IO control, may cause problem with user program?

    IB uses MC to implement Broadcast.
    What does "JOIN" MC mean? Does Join mean create?
    IBTA doesn't have a position on that. Should we tighten this saying purposely left-out?
    Also QKey generation need to be resolved.
    Need to look at spec to find QKey conflict.

    How to define IPOIB interface RUNNING? When?
    before or after join the BC/MC?

    SM Restart losing multicast subscription. IBTA will stress that SM should maintain the info.

    Max UD is max 4K but currently 2K.

    Vivek: status of the drafts.
    3 drafts are being published. Architecture.
    DHCP has been through lastcall, to send note to DHCP group to review it.
    transmission over ip, has been through last call.
    GID format (bit reversed?)has some problem, need to fixed in IB spec.

    Vivek: IPoIB Connected Mode.

    IPOIB modes.
    advantage: Large MTU for better throughput.
    Topic, How to do address resolution in multicast?
    rely on UD QPKey to solve multicast problem.

    Suggested use UD QPN to create a service ID.
    MTU negotiation between min,desired, accepted MTUs.

    on going discussion:
    multiple connections allowed or not?
    one comment: should this be a configuration parameter?

    Vivek -

    Chang - IB Subnet manager MIB status
    Goal is to enable out-of-band configuration of IB
    Subnet Manager and IB fabrics and monitoring of IB fabrics.

    Multicast - suggestion that Multicast table be Read-Write with minimum access of Read. Then it could start out as read only and later if Writes could be allowed without having to introduce a new object. Chang points out that the object might not be the ideal object form for doing that anyway. Issue left open for discussion.

    Trace record - suggested that the author also look at the DisMan draft.

    Notification - why not just use a normal notification macro instead of encapsulating? Wouldn't it be more useful for network managers to use notifications that it can understand. Answer, encapsulation uses the Infiniband notifications. In the current notification, some identifying information such as node number are in the format. This seems sufficient.


    Some Experience with Linux IPoIB Implementations
    IB Subnet Manager MIB
    IPoIB Draft Status