RDMA packages
Contents
Here is a detailed list of the packages that handles RDMA in most popular Linux distributions, separated into categories. There names may be slightly different in different distributions.
Those packages may contain executables, libraries and headers.
Most libraries are separated into several packages:
- Package <library name> - The library files (shared library)
- <library name>-utils - Utilities that come with the library
- It is highly recommended to install this package to any library that is installed. This may come handy when there is a need to debug a problem.
- <library name>-devel - The headers of the library
- One should install it only if he plan to develop programs that uses this library
- <library name>-static - Static library
- One should install it only if he plans to link program that he develops as static
RDMA Programming Interfaces libraries for InfiniBand, RoCE and iWARP
libibverbs
Userspace library which implements the (hardware agnostic) verbs abstraction for using RDMA in software. Most RDMA programs are developed over this library, so it is quite mandatory to install it.
Package Name | Description |
---|---|
libibverbs | libibverbs is a library that allows userspace processes to use RDMA "verbs" as described in the InfiniBand Architecture Specification and the RDMA Protocol Verbs Specification. This includes direct hardware access from userspace to InfiniBand/iWARP adapters (kernel bypass) for fast path operations. |
libibverbs-utils | Useful libibverbs example programs such as ibv_devinfo, which displays information about RDMA devices. |
libibverbs-devel | Header files for the libibverbs library. |
DAPL
This is another library which implements common API for RDMA protocols (it implements the DAT API). However, not a big number of applications/libraries are using it (AFAIK, it is used mostly by specific MPI implementation). One should install it if he needs it.
Package Name | Description |
---|---|
dapl | libdat and libdapl provide a userspace implementation of the DAT 2.0 API and is built to natively support InfiniBand/iWARP network technology. |
dapl-utils | Useful test suites to validate the dapl library API's and operation. |
compat-dapl | The DAT programming API provides a means of utilizing high performance network technologies, such as InfiniBand and iWARP, without needing to write your program to use those technologies directly. This package contains the libraries that implement version 1.2 of the DAT API. The current (and recommended version for any new code) is 2.0. These 1.2 libraries are provided solely for backward compatibility. |
dapl-devel | Header files for libdat and libdapl library. |
Performance
Benchmark tools for RDMA (InfiniBand, RoCE and iWARP) subnets.
Package Name | Description |
---|---|
perftest | Perftest is a collection of simple test programs designed to utilize RDMA communications and provide performance numbers over those RDMA connections. It does not work on normal TCP/IP networks, only on RDMA networks. |
qperf | Measure socket and RDMA performance. |
HW Userspace drivers libraries and tools
Those are the hardware-specific userspace libraries (and tools) for various RDMA devices. One should install the libraries for the hardware that he has in his computers.
Mellanox Technologies
Package Name | Description |
---|---|
libmlx5 | libmlx5 provides a device-specific userspace driver for Mellanox Connect-IB HCAs for use with the libibverbs library. |
libmlx4 | libmlx4 provides a device-specific userspace driver for Mellanox ConnectX HCAs for use with the libibverbs library. |
libmthca | libmthca provides a device-specific userspace driver for Mellanox HCAs (MT23108 InfiniHost and MT25208 InfiniHost III Ex) for use with the libibverbs library. |
mstflint | This package contains a burning tool for Mellanox manufactured HCA cards. It also provides access to the relevant source code. |
Chelsio Communications
Package Name | Description |
---|---|
libcxgb4 | Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio T4 based iWARP capable Ethernet devices. |
libcxgb3 | Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio iWARP capable ethernet devices. |
Emulex
Package Name | Description |
---|---|
libocrdma | libocrdma provides a device-specific userspace driver for Emulex One Command RoCE Adapters for use with the libibverbs library. |
IBM
Package Name | Description |
---|---|
libehca | libehca provides a device-specific userspace driver for IBM HCAs for use with the libibverbs library. |
Intel (formally: NetEffect/QLogic)
Package Name | Description |
---|---|
libnes | Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables NetEffect iWARP capable ethernet devices. |
libipathverbs | libipathverbs provides a device-specific userspace driver for QLogic HCAs for use with the libibverbs library. |
infinipath-psm | The PSM Messaging API, or PSM API, is QLogic's low-level user-level communications interface for the Truescale family of products. PSM users are enabled with mechanisms necessary to implement higher level communications interfaces in parallel environments. |
SoftRoCE
This is a software emulation of the RoCE standard, which can emulate RoCE over any other NIC.
Package Name | Description |
---|---|
librxe | librxe provides a device-specific userspace driver RDMA over Converging Enhanced Ethernet for use with the libibverbs library. |
Subnet Management (InfiniBand only)
Utilities to configure the subnet, you may find them handy if you don't have a Subnet Manager in your switch (i.e. managed switch).
Package Name | Description |
---|---|
opensm | OpenSM is the OpenIB project's Subnet Manager for Infiniband networks. The subnet manager is run as a system daemon on one of the machines in the infiniband fabric to manage the fabric's routing state. This package also contains various tools for diagnosing and testing Infiniband networks that can be used from any machine and do not need to be run on a machine running the opensm daemon. |
ibsim | ibsim provides simulation of infiniband fabric for using with OFA OpenSM, diagnostic and management tools. |
libibmad | libibmad provides low layer IB functions for use by the IB diagnostic and management programs. These include MAD, SA, SMP, and other basic IB functions. |
libibumad | libibumad provides the user MAD library functions which sit on top of the user MAD modules in the kernel. These are used by the IB diagnostic and management tools, including OpenSM. |
libibmad-devel | Development files for the libibmad library. |
libibumad-devel | Development files for the libibumad library. |
opensm-libs | Shared libraries for Infiniband user space access. |
libibcommon | libibcommon provides common utility functions for the OFA diagnostic and management tools. |
Diagnostics (InfiniBand only)
Diagnostics tools for InfiniBand subnet. Using them will help you detect problems in the subnet.
Package Name | Description |
---|---|
infiniband-diags | This package provides IB diagnostic programs and scripts needed to diagnose an IB subnet. |
ibutils | ibutils provides IB network and path diagnostics. |
ibutils-libs | Shared libraries used by the Mellanox Infiniband diagnostic utilities |
Communication Management
Libraries for establishing and tearing down connections between programs.
Communication Management for InfiniBand
Communications management library over InfiniBand addresses.
Package Name | Description |
---|---|
libibcm | libibcm provides a userspace library that handles the majority of the low level work required to open an RDMA connection between two machines. |
libibcm-devel | Development files for the libibcm library. |
Communication Management for RDMA (InfiniBand, RoCE and iWARP)
Communications management library for RDMA (InfiniBand, RoCE and iWARP) with a socket semantics.
Package Name | Description |
---|---|
librdmacm | librdmacm provides a userspace RDMA Communication Management API. |
librdmacm-utils | Example test programs for the librdmacm library. |
librdmacm-devel | Development files for the librdmacm library. |
ibacm | The ib_acm daemon helps reduce the load of managing path record lookups on large InfiniBand fabrics by providing a user space implementation of what is functionally similar to an ARP cache. The use of ib_acm, when properly configured, can reduce the SA packet load of a large IB cluster from O(n^2) to O(n). The ib_acm daemon is started and normally runs in the background, user applications need not know about this daemon as long as their app uses librdmacm to handle connection bring up/tear down. The librdmacm library knows how to talk directly to the ib_acm daemon to retrieve data. |
Userlevel tools for ULPs
User level tools for ULPs (Upper Layer Protocols). One should install them if he uses the ULP that they belong to.
Package Name | Description |
---|---|
srptools | In conjunction with the kernel ib_srp driver, srptools allows you to discover and use SCSI devices via the SCSI RDMA Protocol over InfiniBand. |
rds-tools | Various tools for support of the RDS (Reliable Datagram Socket) API. RDS is specific to InfiniBand and iWARP networks and does not work on non-RDMA. |
Misc
Package Name | Description |
---|---|
rdma | User space initialization scripts for the kernel InfiniBand/iWARP drivers. |
Summary
In this post I tried to cover all the relevant packages for RDMA. If I missed a package that you think that should be added here, please update me.
Comments
Tell us what do you think.
Hi Dotan,
Excellent and useful blog as always. Here are a few packages you may want to consider adding which are in OFED 3.12 and higher.
1. libocrdma - Emulex
2. infinipath-psm - Intel PSM
3. libipathverbs - Intel specific
4. libmlx5 - Mellanox Connect-IB
Thanks - Rupert
Thanks for the complements
:)
I've added all the above packages to the post.
Dotan
Hi,
Thanks for the collection, crisp and more informative..
Please give info about the package "librxe"
Good idea. Done.
In the (soon) future I'll write a post on it.
Thanks
Dotan
Thank you for all the posts in this blog and especially this one.
:)
Thanks!
Dotan
Should libipathverbs (or hfi1verbs) be called a userspace driver? It's just a userspace wrapper around the kernel driver, i.e., its post_send() function just calls ibv_cmd_post_send(), which eventually hits the kernel. Kernel overhead is pretty high for high message rate applications.
I'm curious to know why Intel or QLogic never created a purely userspace verbs driver like Mellanox. AFAIK, Intel and QLogic cards do provide real userspace access via PSM, but PSM is a more heavyweight and restrictive interface than verbs.
Hi.
This is a good question;
the user space library actually doing things in the user space
(whether it is a driver or not - maybe it is a matter of semantics).
RDMA technology allows accessing the device directly from the user space.
However, as you wrote - there are vendors that decided that the actual data path functionality will be in the kernel
because of their engineering decisions
(I'm sorry, but I as a Mellanox employee - I think that this isn't fair for me to give speculations on this).
Thanks
Dotan
Hi Dotan,
First of all your articles are excellent and informative.
I have written a small application on Linux which uses RDMA (Infiniband and RoCE v2) to communicate over Mellanox connect 4/5x driver.
Now I like to port the same application on Windows. Can you give some pointers on how to start with? Btw, I was looking similar SDK for Windows so that I don't need to change RDMA calls. I also looked into winOFED 3.2 for windows and able to compile application as well, but it does not support Mellanox Connect 4/5x.
Thanks
Shrinivas
Hi.
For RDMA windows programming, Microsoft provides an API for Network Direct - this is Windows equivalent for the RDMA verbs.
For more information, you are welcome to read about Network Direct SPI:
https://docs.microsoft.com/en-us/previous-versions/windows/desktop/cc904397(v=vs.85)
Thanks
Dotan