Introduction to Endianness

Let's start by answering the first question: "what is machine Endianness"?

Machine Endianness refer to the way that bytes (i.e. 8 bit values) of word are ordered within memory.

• Bit Endian - the lower bits are saved in memory at the lower addresses. It can be found in PowerPC processors and in networking data. Assuming that the address in memory starts at 0x00100, this is how the value 0x01020304 will look like in memory:
  Address 0x00100:    01
  Address 0x00101:    02
  Address 0x00102:    03
  Address 0x00104:    04
• Little Endian - the higher bits are saved in memory at the lower addresses. It can be found in Intel x86 processors. Assuming that the address in memory starts at 0x00100, this is how the value 0x01020304 will look like in memory:
  Address 0x00100:    04
  Address 0x00101:    03
  Address 0x00102:    02
  Address 0x00104:    01

For more information, one can visit http://en.wikipedia.org/wiki/Endianness.

Packets Endianness

Every packet sent over the wire in RDMA (I completely ignore the transport type since this is relevant for all of them) contain headers. Those headers are in network order (i.e. Big Endian).

Control path and Endianness

Every verb that the user application calls, gets numeric values in variables and structures. Those values should be filled as numeric values in machine Endianness and without any notion of little/Big Endian. It is up to the low-level driver to make the needed translations to make sure that the RDMA device will work in any CPU architecture.

Data path and Endianness

When sending data over the wire using Send/Send with immediate, RDMA Write/RDMA Write with immediate and RDMA Read send the message is sent as a stream of bytes, which don't have any Endianness problem, just like any other networking protocols. If there is a special numeric values meaning in that stream, it is up to the user to take care of it and convert the data to be in network order.

All the data that the user fill in the Send and Receive Requests is in machine Endianness with the following exceptions:

• The immediate data in a Send Request should be in network order.
• The immediate data in a Work Completion is in network order.

Summary

Like many other networking technology, RDMA supports communication between machines with different Endianness. The packets are being sent as Big Endian and it is up to the user to take care of the sent data Endianness - where the data is numeric and not stream of bytes.

Written by: Dotan Barak on January 31, 2014.on February 20, 2015.