ibv_req_notify_cq()
Contents
int ibv_req_notify_cq(struct ibv_cq *cq, int solicited_only); |
Description
ibv_req_notify_cq() requests a Completion Notification on a Completion Queue (CQ).
ibv_req_notify_cq() requests a notification when the next Work Completion of a requested type is added to the CQ. Any Work Completions that existed in the CQ before calling ibv_req_notify_cq() will not result in creating a Completion Notification. The Completion Notification will be read using ibv_get_cq_event().
There are two types of Completion Events that can be requested:
- Solicited Completion Event - Occurs when an incoming Send or RDMA Write with Immediate Data message with the Solicited Event indicator set (i.e. the remote side posted a Send Request with IBV_SEND_SOLICITED set in send_flags) causes a generation of a successful Receive Work Completion or any unsuccessful (either Send of Receive) Work Completion to be added to the CQ.
- Unsolicited Completion Event - occurs when any Work Completion is added to the CQ, whether it is a Send or Receive Work Completion and whether is it successful or unsuccessful Work Completion.
If a Request Completion Notification is pending, i.e. ibv_req_notify_cq() was called and no Completion Notification occurred, subsequent calls to ibv_req_notify_cq() with the same CQ requesting the same Completion Event type will have no effect; only one Completion Notification will be generated. Calling ibv_req_notify_cq() will have an effect only after the Completion Notification will occur.
A Request Completion Notification for the next completion event takes precedence over a Request Completion Notification for a solicited event completion for the same CQ.
If multiple calls to ibv_req_notify_cq() have been made for the same CQ and at least one of the requests set the type to the next completion, a Completion Notification will be generated when the next Completion is added to that CQ (i.e. not for the next Solicited Completion).
Once a Completion Notification occurred, if one wishes to get more Completions Notification, he has to call ibv_req_notify_cq() again.
Parameters
| Name | Direction | Description | ||||
|---|---|---|---|---|---|---|
| cq | in | CQ that was returned from ibv_create_cq() | ||||
| solicited_only | in | Type of the requested Completion Notification.
|
Return Values
| Value | Description |
|---|---|
| 0 | On success |
| errno | On failure. |
| EINVAL | Invalid CQ handle |
Examples
1) Request a notification for the next Completion:
struct ibv_cq *cq; if (ibv_req_notify_cq(cq, 0)) { fprintf(stderr, "Error, ibv_req_notify_cq() failed when requested a " "notification for the next completion\n"); return -1; } |
2) Request a notification for the next Solicited Completion:
struct ibv_cq *cq; if (ibv_req_notify_cq(cq, 1)) { fprintf(stderr, "Error, ibv_req_notify_cq() failed when requested a " "notification for the next solicited completion\n"); return -1; } |
FAQs
I can read Work Completions with ibv_poll_cq(), why would I want to use ibv_req_notify_cq()?
Handling Work Completions with events will decrease the CPU consumption of your application; your process will sleep until a new Work Completion will be added to the CQ.
Can I call ibv_req_notify_cq() with every CQ?
Yes. Although keep in mind that requesting for a notification on Solicited Completions has a meaning only for CQ that is associated with a Receive Queue.
I called ibv_req_notify_cq() when there is a Work Completion in the CQ (of the type that I've asked) will I get a Completion Notification?
No, you won't. A Completion Notification will be generated for the *next* Completion of the type that you've asked (Solicited or next) that will be added to the CQ *after* you called ibv_req_notify_cq().
I called ibv_req_notify_cq() several times before I got a Completion Notification, what is the effect of this?
If all calls to ibv_req_notify_cq() were done when requesting the same Completion type, you will get only one Completion Notification, according to the quested type. If at least one of them requested a Completion Notification for the next Completion, you will get a notification for the next Completion.
Can I ask to get a notification after N Completions will be added to the CQ
No, this isn't supported.
Can I ask to get a notification for every Completion that will add to the CQ automatically (instead of calling ibv_req_notify_cq() again each time)?
No, this isn't supported. After you got a notification, you need to call ibv_req_notify_cq() again if you want to get another notification.
Share Our Posts
Share this post through social bookmarks.
Comments
Tell us what do you think.
Hi.
I hope that this isn't a spam...
But if this is true, thank you
:)
Dotan
Hi Dotan,
I have a few newbie doubts. Can you please explain me the difference between ibv_create_comp_channel() and ibv_req_notify_cq()?
Also, I understand that ibv_get_cq_event() will be required to read the notification event but which of the functions cause the thread to sleep, is it ibv_get_cq_event() or ibv_req_notify_cq()?
Thank you very much,
Regards,
Kaushal
Sure.
ibv_create_comp_channel() creates the object (Completion Event Channel) that will be used to pass the completion events from kernel to user.
ibv_get_cq_event() reads a completion Event from a Completion Event Channel and returns the attribute of the Completion Queue that got this event.
ibv_get_cq_event() is a blocking call by default (however, this behavior can be changed).
ibv_req_notify_cq() only mark that Completion Event is required for a specific Completion Queue.
I hope that I answered...
Dotan
hi Dotan
I am using SEND/RECV, the problem I face is:
I post 10 send quickly. but at the receive side,
only 2 recv CQE will be notified and polled out by the code:( the code is armed to a epoll())
if (ibv_get_cq_event(rc->comp_channel,&evt_cq, &cq_context))
{
printf("ibv get cq event fail, errno:%d\n", errno);
perror(NULL);
return 1;
}
ibv_ack_cq_events(evt_cq, 1);
if (ibv_req_notify_cq(evt_cq, 0))
return 3;
(void)rcDirectPollCq(rc);
but after that,if I directly poll CQ ( without ibv_get_cq_event() ), I can poll the rest 8 CQE out.
How can I make all of them trigger the notify?
Hope some suggest from you.
Thanks
Zhang Yue
Hi Zhang Yue.
I can't see your code,
but I suspect that you have a race and you wait for new events to happen AFTER all the CQEs were arrived.
If you can share the code with me, maybe I'll be able to help you a little bit more.
Thanks
Dotan
Hi Dotan
Thanks for your reply.
I fixed it.
The reason is my polling CQ code did not poll all the recv CQE out, it return early.
I modiry it and make it poll all the CQE out, and test is OK.
Thanks any way.
Zhang Yue.
Hi Dotan.
I know the purpose of the req/get/ack cq calls is to provide a way to be informed when CQE's are available on the CQ, so the CQ can be polled more efficiently, but I need a way to "peek" at the CQE's *before* I poll them from the CQ. I see the ibv_get_notify_cq() call returns a pointer to a cq object that differs from the one I create with ibv_create_cq(). Per the Mellanox documentation, this pointer must be freed by caller. Does that mean there is CQE information on this temporary CQ? Could I call ibv_poll_cq() on the cq returned from ibv_get_notify_cq() to get the CQE events, while still leaving them on the CQ I created to be processed later?
If not, is there any other way to view the CQE's without removing them from the CQ by polling?
Thanks.
Hi.
You can wait to get a completion event for a CQ, and not polling for the Work Completion that caused it.
You cannot peek the Work Completion (i.e. look at its content without dequeuing it from the CQ).
There isn't any actual limitation for this (since the CQE exists in the CQ),
but AFAIK, there isn't any verb that performs this
(this isn't standard; it can be developed as a verb extension by one of the RDMA HW vendors).
Thanks
Dotan