Deployment of an RDMA setup isn't trivial, but it's not rocket science.

In the next few posts, I will describe how to deploy a small RDMA-enabled setup from scratch; from a host point of view. This tutorial will cover deployment a setup for all RDMA protocols: InfiniBand, RoCE and iWARP.

Prerequisite

The smallest setup for using RDMA has two machines. For this setup, a switch is optional, but not mandatory.

The following equipment is needed:

• 2 servers or workstations
• 2 RDMA adapters (either single or dual ports). Here is the list of the RDMA hardware vendors:
  • InfiniBand only:
    • QLogic
  • InfiniBand/RoCE:
    • Mellanox Technologies
  • iWARP
    • Intel (formally: NetEffect)
    • Chelsio Communications
    • Ammasso
• If no switch is used (i.e. back-to-back): 1 network cable
  If there is a switch in the middle:              2 network cables
• Linux-based operating system: RedHat/CentOS, SLES

Note:

In order to use RoCE with an Ethernet switch between the servers, one should configure the switch to prevent packet drops. Instructions that describe how to configure the switch is outside the scope of this post.

Hardware installation

Before installing the needed packages which are required for working with RDMA devices, we first need to install them on the servers. One should follow the next steps:

1. Turn off the computer and unplug the power cord.
2. Open the computer box.
3. Locate an empty PCI/PCI-express for the RDMA device
4. Install the RDMA device in the empty slot.
5. Fasten down the RDMA device to the computer box.
6. Close up the computer and plug-in the power core.
7. Plug in the network cables between the RDMA devices ; In our example, it is preferred to connect them to the same port number in both  devices - for example: port number 1.
8. Turn on the computer.
9. Execute lspci and check that the RDMA device exists in the PCI devices list. For example:

# lspci | grep -i Mellanox
 
04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

For more information on how to install PCI/PCI express adapter in the computer, I'm sure you can find a video on YouTube that can help ...

In the next posts, I'll cover the installation of RDMA packages in the popular Linux-based operating systems (inbox and OFED based distributions).

Written by: Dotan Barak on August 18, 2014.on February 23, 2015.