Loopback messages in RDMA

What is a loopback message?

A loopback message is a message that was sent in an RDMA device, in a specific port from one QP to another/same QP on that device at the same port; the destination address of the address vector (in the QP or the AH) is equal to the address of the source port: LID and GID (if used).

If the used destination address attributes differ from the address of the source port, even if both ports belong to the same RDMA device - this message isn't considered a loopback message.

What will happen in a loopback message?

Unlike other protocols (for example, Ethernet), where the loopback of a message is handled by software (i.e., network stack), the device isn't aware of the existence of that message. In RDMA, the RDMA device processes all the Work Requests. It will fetch the Work Request, process it, fetch the needed data, and store it, ignoring the fact that this is a loopback message, and at the end, generate Work Completion(s) - if required.

So, what is the difference between loopback and non-loopback messages?

In loopback messages, packets won't be generated and won't appear at the port's physical link, and this has several effects:

• Packet-sniffer outside this port won't see loopback messages.
• Port counters won't be incremented for loopback messages (number of bytes + number of packets)
• The RDMA device may ignore path attributes (MTU, source path bits, static rate). The SL can be used - this is implementation dependent.
• The RDMA device may have optimization for handling loopback messages since they don't need to be partitioned into packets, create headers, calculate checksums, and validate them when received.

Usage for loopback messages

The software doesn't have to be aware that a message is a loopback message;

i.e., if a specific application has several processes running on several machines, part of them may run on the same machine, and it doesn't have to write different code for loopback messages. The RDMA protocol can handle it transparently.

Loopback messages can be a fast way to copy big chunks of data in the same machine (and even in the same process) asynchronicity. memcpy() is good, but DMA using a different engine than the CPU may be better, leaving the CPU to do other things...