Working with RDMA using OFED
Contents
OFED (OpenFabrics Enterprise Distribution) is a package that developed and released by the OpenFabrics Alliance (OFA), as a joint effort of many companies that are part of the RDMA scene. It contains the latest upstream software packages (both kernel modules and userspace code) to work with RDMA. It supports InfiniBand, Ethernet and RoCE transports.
This package supports most major Linux distributions and CPU architectures.
Installing RDMA packages
Prerequisites
Make sure that there is at least 1 GB free space in your disk.
Download the package
Open the following URL in a web browser:
https://www.openfabrics.org/downloads/
Click the "OFED" directory and then click on the directory of the OFED version that you wish to download. Download the .tgz file of the release from that directory (it is highly advised to download the final released package and not one of the RCs).
Installation
Pre-install
After downloading the tgz file, extract it and enter the directory that the files exist in:
[root@localhost]# cd /tmp/OFED
[root@localhost]# tar xzf OFED-3.12.tgz
[root@localhost]# cd OFED-3.12
Post-install
Delete the generated directory that holds the extracted files:
[root@localhost]# rm -fr /tmp/OFED
Now, everything is ready and installation of OFED can be started.
Installation
The installation script will:
- Verify that installed kernel is supported.
- Uninstall all installed RDMA packages, which are part of the Linux distribution or part of any RDMA package.
- Build the OFED binary RPMs from the SRPMs.
- Install the binary RPMs.
- Fix local configuration files, if needed.
The same installation procedure is used in all Linux distributions.
The following command line will start the installation process.
Now, a textual menu will be displayed:
OFED Distribution Software Installation Menu
1) View OFED Installation Guide
2) Install OFED Software
3) Show Installed Software
4) Configure IPoIB
5) Uninstall OFED Software
Q) Exit
Select Option [1-5]:
Choose "2" to go to the installation section.
OFED Distribution Software Installation Menu
1) Basic (OFED modules and basic user level libraries)
2) HPC (OFED modules and libraries, MPI and diagnostic tools)
3) All packages (all of Basic, HPC)
4) Customize
Q) Exit
Select Option [1-4]:
There are 4 options to decide which packages will be built (i.e. binary RPMs will be built from the SRPMs) and installed:
- Install Basic packages which include kernel modules and the basic userspace libraries and tools: low-level drivers libraries and tools for all RDMA devices, libibverbs and librdmacm.
- Install HPC packages which include all the Basic package + DAPL, subnet diagnostics tools, performance benchmarks and MPI packages.
- Install all the available packages.
- Customize the packages to be installed.
One should decide which option is the best for him.
Building all the binaries RPMs may take several minutes.
In the following example screenshot, I choose "1" (Basic).
(some may have been added by the installer due to package dependencies):ofed-scripts
libibverbs
libibverbs-utils
libibverbs-devel
libmthca
libmlx4
libmlx5
libcxgb3
libcxgb4
libnes
librdmacm
librdmacm-utils
libocrdma
ofed-docs
compat-rdmaBuild ofed-scripts RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/ofed-scripts-3.12-0.1.g751417d.src.rpm
Install ofed-scripts RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/ofed-scripts-3.12-0.1.g751417d.i586.rpm
Build libibverbs RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libibverbs-1.1.7-1.src.rpm
Install libibverbs RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libibverbs-1.1.7-1.i586.rpm
Install libibverbs-utils RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libibverbs-utils-1.1.7-1.i586.rpm
Install libibverbs-devel RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libibverbs-devel-1.1.7-1.i586.rpm
Build libmthca RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libmthca-1.0.6-1.src.rpm
Install libmthca RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libmthca-1.0.6-1.i586.rpm
Build libmlx4 RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' --nodeps /tmp/OFED/OFED-3.12/SRPMS/libmlx4-1.0.5-1.src.rpm
Install libmlx4 RPM:
Running rpm -iv --nodeps /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libmlx4-1.0.5-1.i586.rpm
Build libmlx5 RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libmlx5-1.0.1-1.src.rpm
Install libmlx5 RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libmlx5-1.0.1-1.i586.rpm
Build libcxgb3 RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libcxgb3-1.3.1-1.src.rpm
Install libcxgb3 RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libcxgb3-1.3.1-1.i586.rpm
Build libcxgb4 RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libcxgb4-1.3.3-1.src.rpm
Install libcxgb4 RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libcxgb4-1.3.3-1.i586.rpm
Build libnes RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libnes-1.1.4-0..src.rpm
Install libnes RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libnes-1.1.4-0..i586.rpm
Build librdmacm RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/librdmacm-1.0.18.1-1.src.rpm
Install librdmacm RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/librdmacm-1.0.18.1-1.i586.rpm
Install librdmacm-utils RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/librdmacm-utils-1.0.18.1-1.i586.rpm
Build mstflint RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/mstflint-3.6.0-1.8.g7d4dede.src.rpm
Install mstflint RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/mstflint-3.6.0-1.8.g7d4dede.i586.rpm
Build libocrdma RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/libocrdma-1.0.2-1.src.rpm
Install libocrdma RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/libocrdma-1.0.2-1.i586.rpm
Build ofed-docs RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target i586 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /tmp/OFED/OFED-3.12/SRPMS/ofed-docs-3.12-0.1.ge933a19.src.rpm
Install ofed-docs RPM:
Running rpm -iv /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/ofed-docs-3.12-0.1.ge933a19.i586.rpm
Build compat-rdma RPM
Running rpmbuild --rebuild --define '_topdir /var/tmp//OFED_topdir' --define '_suse_os_install_post %{nil}' --nodeps --define '_dist .sles11sp3' --define 'configure_options --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx5-mod --with-cxgb3-mod --with-cxgb4-mod --with-nes-mod --with-ocrdma-mod --with-ipoib-mod' --define 'build_kernel_ib 1' --define 'build_kernel_ib_devel 1' --define 'KVERSION 3.0.76-0.11-pae' --define 'K_SRC /lib/modules/3.0.76-0.11-pae/build' --define '_release 0.1.gac916cb.3.0.76_0.11_pae' --define 'network_dir /etc/sysconfig/network' --define '_prefix /usr' --define '__arch_install_post %{nil}' /tmp/OFED/OFED-3.12/SRPMS/compat-rdma-3.12-0.1.gac916cb.src.rpm
Install compat-rdma RPM:
Running rpm -iv --nodeps /tmp/OFED/OFED-3.12/RPMS/sles-release-11.3-1.138/i686/compat-rdma-3.12-0.1.gac916cb.3.0.76_0.11_pae.i586.rpmThe default IPoIB interface configuration is based on DHCP.
Note that a special patch for DHCP is required for supporting IPoIB.
The patch is available under docs/dhcp
If you do not have DHCP, you must change this configuration in the following steps.
IPoIB interfaces configured successfully
Press any key to continue ...Installation finished successfully.
Press any key to continue...
Different versions of OFED or different Linux distributions may have different RPM names and versions.
Reboot is usually required to complete the installation.
Notes
Failure in uninstalling automatically an RPM
If there is a problem to uninstall any RPM, the installer will print an error message and the user will need to remove this RPM manually.
Missing RPMs during the binary RPMs build
Before building the binary RPMs, the installer will verify that all required RPMs are installed. If any RPM (according to the Linux distribution and the list of packages to install) is missing, an error message which specifies the name of the missing RPM and the package that needs it will be displayed. For example:
libstdc++43-devel rpm is required to build mstflint
gcc-c++ rpm is required to build mstflint
One should install the missing RPMs and restart the installation.
Deployment of OFED in a cluster
A configuration file with the name ofed.conf will be saved locally. It will describe the chosen options. If one wishes to install OFED automatically with the same options that were taken, executing the installer with -c ofed.conf will do the trick.
The built binary RPMs are saved in the installation directory, under the RPMS directory.
When installing the OFED release in a cluster, it is a good practice installing it on a single machine manually and after its completion, take the whole installation directory (with the configuration file and binary RPMs) and install it in the rest of the machines automatically.
Uninstalling RDMA packages
This package comes with an uninstall script that will clean everything. The following command line will uninstall the OFED packages:
/usr/sbin/ofed_uninstall.shThis program will uninstall all OFED packages on your machine.Do you want to continue?[y/N]:
This "yes/no" question should be answered with "y". Only pressing the "enter" key will use the default (N) and the uninstallation will be terminated.
Running /usr/sbin/vendor_pre_uninstall.sh
Removing OFED Software installations
Running /bin/rpm -e --allmatches compat-rdma libibverbs libibverbs-devel libibverbs-utils libmthca libmlx4 libmlx5 libcxgb3 libcxgb4 libnes libibumad libibumad-devel librdmacm librdmacm-utils librdmacm-devel libocrdma perftest dapl dapl-devel dapl-devel-static dapl-utils qperf ofed-docs ofed-scripts
warning: /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsave
Running /tmp/24985-ofed_vendor_post_uninstall.sh
Uninstall can also be done from the installation menu (Option "5" in the first menu) or using the uninstall.sh, which provided in the OFED package.
Starting the RDMA services
Load the RDMA drivers using the following command line:
If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:
By default, when installing the RDMA packages, the RDMA service will be loaded automatically when the operating system is loaded. If this doesn't happen, one should fix manually.
Stopping the RDMA services
If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line:
Unload the RDMA drivers using the following command line:
RDMA configuration file(s)
1. The openibd service loads the configuration file: /etc/infiniband/openib.conf. This file controls which modules will be loaded during the service startup and some attributes about the RDMA modules. The following parameters are supported:
| Parameter name | Description | Supported values |
|---|---|---|
| ONBOOT | Start RDMA modules/drivers upon machine boot | yes/no |
| NODE_DESC | Node description prefix command | Command string. For example: $(hostname -s) |
| NODE_DESC_TIME_BEFORE_UPDATE | Time, in seconds, to wait before updating node description | Non-negative integer numbers |
| RUN_SYSCTL | Run sysctl performance tuning on network attributes script | yes/no |
| RDMA_CM_LOAD | Load RDMA_CM module | yes/no |
| RDMA_UCM_LOAD | Load RDMA_UCM module | yes/no |
| UCM_LOAD | Load UCM module | yes/no |
| RENICE_IB_MAD | Increase ib_mad thread priority | yes/no |
| IPOIB_LOAD | Load IPoIB module | yes/no |
| SET_IPOIB_CM | Enable IPoIB Connected Mode | yes/no |
| MLX5_LOAD | Load MLX5 driver | yes/no |
| MLX4_LOAD | Load MLX4 driver | yes/no |
| MLX4_EN_LOAD | Load MLX4_EN driver | yes/no |
| MTHCA_LOAD | Load MTHCA driver | yes/no |
| CXGB4_LOAD | Load CXGB4 driver | yes/no |
| CXGB3_LOAD | Load CXGB3 driver | yes/no |
| NES_LOAD | Load iw_nes driver | yes/no |
| OCRDMA_LOAD | Load OCRDMA driver | yes/no |
2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines, if they weren't added by the installation script:
* soft memlock unlimited
* hard memlock unlimited
This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.
After login again, executing the following command line will print how much memory (in KB) can be locked:
(the expected output is: "unlimited").
- If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.
Share Our Posts
Share this post through social bookmarks.
Comments
Tell us what do you think.
Hi,
What could be a reason of "device's health compromised" kernel message? I see it appears here: http://lxr.free-electrons.com/source/drivers/net/ethernet/mellanox/mlx5/core/health.c
but can't realise when it might be triggered.
Thanks!
just a followup: the error string is "ICM fetch PCI error".
I would suggest to update the driver and firmware.
And if this didn't solve the problem - contact Mellanox support.
Thanks
Dotan