Skip to content

RDMA packages

Contents

4.00 avg. rating (83% score) - 3 votes

Here is a detailed list of the packages that handles RDMA in most popular Linux distributions, separated into categories. There names may be slightly different in different distributions.

Those packages may contain executables, libraries and headers.
Most libraries are separated into several packages:

  • Package <library name> - The library files (shared library)
  • <library name>-utils - Utilities that come with the library
    • It is highly recommended to install this package to any library that is installed. This may come handy when there is a need to debug a problem.
  • <library name>-devel - The headers of the library
    • One should install it only if he plan to develop programs that uses this library
  • <library name>-static - Static library
    • One should install it only if he plans to link program that he develops as static

RDMA Programming Interfaces libraries for InfiniBand, RoCE and iWARP

libibverbs

Userspace library which implements the (hardware agnostic) verbs abstraction for using RDMA in software. Most RDMA programs are developed over this library, so it is quite mandatory to install it.

Package Name Description
libibverbs libibverbs is a library that allows userspace processes to use RDMA "verbs" as described in the InfiniBand Architecture Specification and the RDMA Protocol Verbs Specification. This includes direct hardware access from userspace to InfiniBand/iWARP adapters (kernel bypass) for fast path operations.
libibverbs-utils Useful libibverbs example programs such as ibv_devinfo, which displays information about RDMA devices.
libibverbs-devel Header files for the libibverbs library.

DAPL

This is another library which implements common API for RDMA protocols (it implements the DAT API). However, not a big number of applications/libraries are using it (AFAIK, it is used mostly by specific MPI implementation). One should install it if he needs it.

Package Name Description
dapl libdat and libdapl provide a userspace implementation of the DAT 2.0 API and is built to natively support InfiniBand/iWARP network technology.
dapl-utils Useful test suites to validate the dapl library API's and operation.
compat-dapl The DAT programming API provides a means of utilizing high performance network technologies, such as InfiniBand and iWARP, without needing to write your program to use those technologies directly. This package contains the libraries that implement version 1.2 of the DAT API. The current (and recommended version for any new code) is 2.0. These 1.2 libraries are provided solely for backward compatibility.
dapl-devel Header files for libdat and libdapl library.

Performance

Benchmark tools for RDMA (InfiniBand, RoCE and iWARP) subnets.

Package Name Description
perftest Perftest is a collection of simple test programs designed to utilize RDMA communications and provide performance numbers over those RDMA connections. It does not work on normal TCP/IP networks, only on RDMA networks.
qperf Measure socket and RDMA performance.

HW Userspace drivers libraries and tools

Those are the hardware-specific userspace libraries (and tools) for various RDMA devices. One should install the libraries for the hardware that he has in his computers.

Mellanox Technologies

Package Name Description
libmlx5 libmlx5 provides a device-specific userspace driver for Mellanox Connect-IB HCAs for use with the libibverbs library.
libmlx4 libmlx4 provides a device-specific userspace driver for Mellanox ConnectX HCAs for use with the libibverbs library.
libmthca libmthca provides a device-specific userspace driver for Mellanox HCAs (MT23108 InfiniHost and MT25208 InfiniHost III Ex) for use with the libibverbs library.
mstflint This package contains a burning tool for Mellanox manufactured HCA cards. It also provides access to the relevant source code.

Chelsio Communications

Package Name Description
libcxgb4 Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio T4 based iWARP capable Ethernet devices.
libcxgb3 Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio iWARP capable ethernet devices.

Emulex

Package Name Description
libocrdma libocrdma provides a device-specific userspace driver for Emulex One Command RoCE Adapters for use with the libibverbs library.

IBM

Package Name Description
libehca libehca provides a device-specific userspace driver for IBM HCAs for use with the libibverbs library.

Intel (formally: NetEffect/QLogic)

Package Name Description
libnes Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables NetEffect iWARP capable ethernet devices.
libipathverbs libipathverbs provides a device-specific userspace driver for QLogic HCAs for use with the libibverbs library.
infinipath-psm The PSM Messaging API, or PSM API, is QLogic's low-level user-level communications interface for the Truescale family of products. PSM users are enabled with mechanisms necessary to implement higher level communications interfaces in parallel environments.

SoftRoCE

This is a software emulation of the RoCE standard, which can emulate RoCE over any other NIC.

Package Name Description
librxe librxe provides a device-specific userspace driver RDMA over Converging Enhanced Ethernet for use with the libibverbs library.

Subnet Management (InfiniBand only)

Utilities to configure the subnet, you may find them handy if you don't have a Subnet Manager in your switch (i.e. managed switch).

Package Name Description
opensm OpenSM is the OpenIB project's Subnet Manager for Infiniband networks. The subnet manager is run as a system daemon on one of the machines in the infiniband fabric to manage the fabric's routing state. This package also contains various tools for diagnosing and testing Infiniband networks that can be used from any machine and do not need to be run on a machine running the opensm daemon.
ibsim ibsim provides simulation of infiniband fabric for using with OFA OpenSM, diagnostic and management tools.
libibmad libibmad provides low layer IB functions for use by the IB diagnostic and management programs. These include MAD, SA, SMP, and other basic IB functions.
libibumad libibumad provides the user MAD library functions which sit on top of the user MAD modules in the kernel. These are used by the IB diagnostic and management tools, including OpenSM.
libibmad-devel Development files for the libibmad library.
libibumad-devel Development files for the libibumad library.
opensm-libs Shared libraries for Infiniband user space access.
libibcommon libibcommon provides common utility functions for the OFA diagnostic and management tools.

Diagnostics (InfiniBand only)

Diagnostics tools for InfiniBand subnet. Using them will help you detect problems in the subnet.

Package Name Description
infiniband-diags This package provides IB diagnostic programs and scripts needed to diagnose an IB subnet.
ibutils ibutils provides IB network and path diagnostics.
ibutils-libs Shared libraries used by the Mellanox Infiniband diagnostic utilities

Communication Management

Libraries for establishing and tearing down connections between programs.

Communication Management for InfiniBand

Communications management library over InfiniBand addresses.

Package Name Description
libibcm libibcm provides a userspace library that handles the majority of the low level work required to open an RDMA connection between two machines.
libibcm-devel Development files for the libibcm library.

Communication Management for RDMA (InfiniBand, RoCE and iWARP)

Communications management library for RDMA (InfiniBand, RoCE and iWARP) with a socket semantics.

Package Name Description
librdmacm librdmacm provides a userspace RDMA Communication Management API.
librdmacm-utils Example test programs for the librdmacm library.
librdmacm-devel Development files for the librdmacm library.
ibacm The ib_acm daemon helps reduce the load of managing path record lookups on large InfiniBand fabrics by providing a user space implementation of what is functionally similar to an ARP cache. The use of ib_acm, when properly configured, can reduce the SA packet load of a large IB cluster from O(n^2) to O(n). The ib_acm daemon is started and normally runs in the background, user applications need not know about this daemon as long as their app uses librdmacm to handle connection bring up/tear down. The librdmacm library knows how to talk directly to the ib_acm daemon to retrieve data.

Userlevel tools for ULPs

User level tools for ULPs (Upper Layer Protocols). One should install them if he uses the ULP that they belong to.

Package Name Description
srptools In conjunction with the kernel ib_srp driver, srptools allows you to discover and use SCSI devices via the SCSI RDMA Protocol over InfiniBand.
rds-tools Various tools for support of the RDS (Reliable Datagram Socket) API. RDS is specific to InfiniBand and iWARP networks and does not work on non-RDMA.

Misc

Package Name Description
rdma User space initialization scripts for the kernel InfiniBand/iWARP drivers.

Summary

In this post I tried to cover all the relevant packages for RDMA. If I missed a package that you think that should be added here, please update me.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Rupert Dance says: August 24, 2014

    Hi Dotan,

    Excellent and useful blog as always. Here are a few packages you may want to consider adding which are in OFED 3.12 and higher.

    1. libocrdma - Emulex
    2. infinipath-psm - Intel PSM
    3. libipathverbs - Intel specific
    4. libmlx5 - Mellanox Connect-IB

    Thanks - Rupert

    • Dotan Barak says: August 24, 2014

      Thanks for the complements
      :)

      I've added all the above packages to the post.

      Dotan

  2. Dinesh.K.B says: May 7, 2015

    Hi,
    Thanks for the collection, crisp and more informative..
    Please give info about the package "librxe"

    • Dotan Barak says: May 7, 2015

      Good idea. Done.

      In the (soon) future I'll write a post on it.

      Thanks
      Dotan

  3. Raghavendra Talur says: July 27, 2016

    Thank you for all the posts in this blog and especially this one.

    • Dotan Barak says: July 29, 2016

      :)

      Thanks!
      Dotan

  4. Anuj Kalia says: July 6, 2017

    Should libipathverbs (or hfi1verbs) be called a userspace driver? It's just a userspace wrapper around the kernel driver, i.e., its post_send() function just calls ibv_cmd_post_send(), which eventually hits the kernel. Kernel overhead is pretty high for high message rate applications.

    I'm curious to know why Intel or QLogic never created a purely userspace verbs driver like Mellanox. AFAIK, Intel and QLogic cards do provide real userspace access via PSM, but PSM is a more heavyweight and restrictive interface than verbs.

    • Dotan Barak says: July 21, 2017

      Hi.

      This is a good question;
      the user space library actually doing things in the user space
      (whether it is a driver or not - maybe it is a matter of semantics).

      RDMA technology allows accessing the device directly from the user space.
      However, as you wrote - there are vendors that decided that the actual data path functionality will be in the kernel
      because of their engineering decisions
      (I'm sorry, but I as a Mellanox employee - I think that this isn't fair for me to give speculations on this).

      Thanks
      Dotan

  5. Shrinivas Dudhani says: June 16, 2020

    Hi Dotan,

    First of all your articles are excellent and informative.
    I have written a small application on Linux which uses RDMA (Infiniband and RoCE v2) to communicate over Mellanox connect 4/5x driver.

    Now I like to port the same application on Windows. Can you give some pointers on how to start with? Btw, I was looking similar SDK for Windows so that I don't need to change RDMA calls. I also looked into winOFED 3.2 for windows and able to compile application as well, but it does not support Mellanox Connect 4/5x.

    Thanks
    Shrinivas

Add a Comment

This comment will be moderated; answer may be provided within 14 days.

Time limit is exhausted. Please reload CAPTCHA.