By default, all Work Requests generate Work Completions when their processing is finished. However, Send Request may or may not generate Work Completions when their processing is finished - this is fully controllable by the application and this is called Unsignaled Completions. In this post, I will cover when and how to use it.

What is Unsignaled Completion?

Unsignaled Completion is a mechanism that allows the application to post Send Requests that when their processing ends, Work Completions won't be generated in the Completion Queue, which is associated with the Send Queue as long as the Work Completion processing ends without an error. If a Send Request ended with an error, it will generate a Work Completion even if it was posted to use Unsignaled Completion.

Why use Unsignaled Completion?

Handling Work Requests consumes machine's resources:

• Generation of the Work Completion by the RDMA device
• Polling for the Completion Queue for Work Completions
• Reading the Work Completion and check it status

Handling less Work Completions will help to reduce the CPU usage of the application, thus to increase the effectiveness of the application and to improve the latency of handled messages.

When to use Unsignaled Completion?

Unlike Work Completions that are generated from incoming messages that consume Receive Requests in asynchronous way, Work Requests that generated from Send Requests that their processing ended is quite expected by the application. Furthermore, Work Completions of finished Receive Requests may contain important information (opcode, message size, origin, etc.) but Work Completions of finished Send Requests doesn't contain important information (expect for information that is encoded in the wr_id, which is fully controlled by the application.

There are two groups of operations that can be used in Send Requests:

1. Operations that request the remote side to send data (RDMA Read and Atomic operations)
2. Operations that send data to the remote side (Send and RDMA Write operations with or without immediate data)

Unsignaled Completions can be used for operations from the first group if all the following conditions are met:

• The application can request the remote side to read several buffers before it needs to use their content
• All incoming data is written to different memory buffers
• The Memory Regions that contain the incoming data don't need to be destroyed/reused

Unsignaled Completions can be used for operations from the second group if all the following conditions are met:

• The application can send several messages before it needs acknowledge that there processing ended
• The Memory Regions that contain the data to be sent don't need to be destroyed/reused
• For UD QPs, Address Handles to the remote side don't need to be destroyed/reused

How to use Unsignaled Completion?

The support of Unsignaled Completion is configured per Queue Pair. When creating the Queue Pair, it should be created to support Unsignaled Completions in the Send Queue (the attribute qp_init_attr.sq_sig_all should be set to zero).

For every posted Send Request, if the flag IBV_SEND_SIGNALED is set in wr.send_flags then a Work Completion will be generated when this Send Request processing is ended. If this flags is cleared, then a Work Completion won't be generated (as long as it completed without an error).

Gotchas and Pitfalls

All posted Send Requested, Signaled and Unsignaled, are considered outstanding until a Work Completion that they, or Send Requests that were posted after them, was polled from the Completion Queue associated with the Send Queue. This means if one works with a Queue Pair that was configured to work with Unsignaled Completions, he must make sure that occasionally (before the Send Queue is full with outstanding Send Requests) a Send Request that generate Work Completion will be posted.

Not following this rule may lead to a case that the Send Queue is full with Send Requests that won't generate Work Completion:

• The Send Queue is full, so no new Send Requests can be posted to it
• The Send Queue can't be emptied, since no Work Completion can be generated anymore (the reason is that no Work Completion, that can generate Work Completion that polling it will empty the Send Queue, can be posted)
• The status of all posted Send Request is considered unknown

Examples

1. Prepare a QP init attributes with an explicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr = {
	.send_cq = cq,
	.recv_cq = cq,
	.cap     = {
		.max_send_wr = 10,
		.max_recv_wr = 10,
		.max_send_sge = 1,
		.max_recv_sge = 1
	},
	.qp_type    = IBV_QPT_RC,
	.sq_sig_all = 0
};

2. Prepare a QP init attributes with an implicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr = {
	.send_cq = cq,
	.recv_cq = cq,
	.cap     = {
		.max_send_wr = 10,
		.max_recv_wr = 10,
		.max_send_sge = 1,
		.max_recv_sge = 1
	},
	.qp_type    = IBV_QPT_RC
};

3. Prepare a QP init attributes with an implicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr;
 
memset(&init_attr, 0, sizeof(init_attr));
 
init_attr.send_cq = cq;
init_attr.recv_cq = cq;
init_attr.cap.max_send_wr = 10;
init_attr.cap.max_recv_wr = 10;
init_attr.cap.max_send_sge = 1;
init_attr.cap.max_recv_sge = 1;
init_attr.qp_type = IBV_QPT_RC;

4. Prepare Send Request that ends with a Work Completion in a Queue Pair which configured to support Unsignaled Completions:

struct ibv_send_wr wr = {
	.num_sge    = 0,
	.opcode     = IBV_WR_SEND,
	.send_flags = IBV_SEND_SIGNALED
};

5. Prepare Send Request that won't end with a Work Completion in a Queue Pair which configured to support Unsignaled Completions:

struct ibv_send_wr wr = {
	.num_sge    = 0,
	.opcode     = IBV_WR_SEND,
	.send_flags = 0
};

FAQs

I accidentally filled the Send Queue with Send Requests that use Unsignaled Completion and I can't post anymore Send Requests to the Queue Pair. How can I recover from this state?

You cannot recover from this state. You need to destroy this Queue Pair and create a new one.

Written by: Dotan Barak on June 30, 2014.on February 22, 2015.