The OpenFabrics Enterprise Distribution (OFED) is a package for Linux that includes all of the needed software (libraries, utilities and drivers) in order to work with RDMA enabled protocols (InfiniBand and iWARP).

OFED content

This package includes many components to allow the support of RDMA. Here is a diagram, published by the OpenFabrics, that describes the software layers:

Applications and utilities

• Diagnostics tools - Tools for debugging and diagnostics in an InfiniBand fabric
• Open Subnet Manager (OpenSM) - Program that manages and configures an InfiniBand fabric
• Various Message Passing Interface (MPI) packages - API for development of applications that can be executed in parallel on machine machines. Those packages support many interconnect technologies, including InfiniBand

User level APIs

APIs that allow development of RDMA enabled applications at the user level.

• RDMA verbs - API of the verbs interface that was defined in the InfiniBand and iWARP specifications
• uDAPL - API that was defined by the DAT collaborative
• RDMA Communication Manager (RDMA-CM) - libraries for setting up RDMA connections

Upper Level Protocols (ULPs)

Modules that allow upper level and legacy applications to work over RDMA.

• IP over InfiniBand (IPoIB) - A module that encapsulates IP over InfiniBand
• Socket Direct Protocol (SDP) - A module that implements TCP sockets over InfiniBand
• Reliable Datagram Sockets (RDS) - A module that allows high-performance, low latency reliable connectionless protocol for delivering datagrams
• SCSI RDMA Protocol (SRP) - A module that allows one computer to access SCSI devices attaches to another computer over RDMA
• iSCSI Extensions for RDMA (iSER) - A modules that allows one computer to access iSCSI devices attaches to another computer over RDMA

Mid layers

Kernel modules that enables the RDMA support for both user and kernel level.

• IB core - The RDMA core module for the RDMA implementation in the kernel
• CMA - The communication manager module
• MAnagement Datagram (MAD) - Module for handling MADs in the InfiniBand fabric
• Subnet Administrator (SA) client - Module for handling communication with the SA

Providers

Low level driver for the RDMA devices (HCAs and RNICs) of various vendors, for example:

• Mellanox Technologies
• Qlogic
• Chelsio Communications
• IBM
• Intel

General

OFED is being developed, tested and released as a joint effort of many companies and organizations. It being released every six months and supports various RDMA devices and Linux distributions.

Links:

https://www.openfabrics.org/ofed-for-linux-ofed-for-windows/ofed-overview.html

http://en.wikipedia.org/wiki/OpenFabrics_Alliance

Written by: Dotan Barak on April 25, 2012.on February 16, 2015.