There are many code samples on the internet that explain how to send messages using the SEND operation over Reliable Connected (RC) QP. In this post I will explain how to convert the code from this kind of example to support other features (such as different QP transport types or different opcodes).

Converting RC Send to UC Send

Converting a code that perform Send over RC QP to Send over UC QP requires the following modifications:

• If the QP is associated with an SRQ, make sure that the RDMA device that you work with supports it (since not all RDMA devices support associating UC QPs with an SRQ)
• When creating the QP, replace IBV_QPT_RC with IBV_QPT_UC
• When modifying the QP state from Init->RTR, the following masks/attributes aren't supported: IBV_QP_MAX_DEST_RD_ATOMIC and IBV_QP_MIN_RNR_TIMER
• When modifying the QP state from RTR->RTS, the following masks/attributes aren't supported: IBV_QP_TIMEOUT, IBV_QP_RETRY_CNT, IBV_QP_RNR_RETRY, IBV_QP_MAX_QP_RD_ATOMIC
• If your code assumes that there is an RNR retry, you should stop using this assumption
• If there is a missing message, there is a retransmission flow that should be prevented

Converting RC Send to unicast UD Send

Converting a code that perform Send over RC QP to unicast Send over UD QP requires the following modifications:

• When creating the QP, replace IBV_QPT_RC with IBV_QPT_UD
• When modifying the QP state from Reset->Init, the following masks/attributes aren't supported: IBV_QP_ACCESS_FLAGS. The following masks/attributes are supported IBV_QP_QKEY
• When modifying the QP state from Init->RTR, the following masks/attributes aren't supported: IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER
• When modifying the QP state from RTR->RTS, the following masks/attributes aren't supported: IBV_QP_TIMEOUT, IBV_QP_RETRY_CNT, IBV_QP_RNR_RETRY, IBV_QP_MAX_QP_RD_ATOMIC
• Address Handle (AH) should be created
• If your code assumes that there is an RNR retry, you should stop using this assumption
• If there is a missing message, there is a retransmission flow that should be prevented
• Message size is limited to the minimum path MTU between the requestor and the responder
• When posting a SR, the attributes in wr.ud.* should be filled
• When posting a RR, an extra 40 bytes should be added (to hold the GRH)
• The data that was received by an UD QP starts at offset 40 from the beginning of the buffer (after the reserved space to the GRH)

Converting RC Send to RC RDMA Write

Converting a code that perform Send over RC QP to RDMA Write over RC QP requires the following modifications:

• When registering the MR in the responder side, the mask IBV_ACCESS_REMOTE_WRITE should be enabled
• When modifying the QP state from Reset->Init, the mask IBV_ACCESS_REMOTE_WRITE should be enabled in attr->qp_access_flags
• The responder must send to the requestor the remote address that he should access, the length of the buffer and the rkey
• When posting a SR, the attributes in wr.rdma.* should be filled
• No need to post RRs in the responder side, unless the opcode RDMA Write with immediate is being used

Converting RC Send to RC RDMA Read

Converting a code that perform Send over RC QP to RDMA Read over RC QP requires the following modifications:

• When registering the MR in the responder side, the mask IBV_ACCESS_REMOTE_READ should be enabled
• When modifying the QP state from Reset->Init, the mask IBV_ACCESS_REMOTE_READ should be enabled in attr->qp_access_flags
• When modifying the QP state from Init->RTR the value in attr->max_dest_rd_atomic shouldn't be zero
• When modifying the QP state from RTR->RTS the value in attr->max_rd_atomic shouldn't be zero
• The responder must send to the requestor the remote address that he should access, the length of the buffer and the rkey
• When posting a SR, the attributes in wr.rdma.* should be filled
• No need to post RRs in the responder side

Converting RC Send to RC Atomic operation

Converting a code that perform Send over RC QP to Atomic operations over RC QP requires the following modifications:

• When registering the MR in the responder side, the mask IBV_ACCESS_REMOTE_ATOMIC should be enabled
• When modifying the QP state from Reset->Init, the mask IBV_ACCESS_REMOTE_ATOMIC should be enabled in attr->qp_access_flags
• When modifying the QP state from Init->RTR the value in attr->max_dest_rd_atomic shouldn't be zero
• When modifying the QP state from RTR->RTS the value in attr->max_rd_atomic shouldn't be zero
• The responder must send to the requestor the remote address that he should access, the length of the buffer and the rkey
• Message size is limited to 8 bytes
• When posting a SR, the attributes in wr.atomic.* should be filled
• The remote address to access must be 8 bytes aligned
• No need to post RRs in the responder side

Written by: Dotan Barak on June 29, 2013.on March 9, 2019.