ibv_create_comp_channel()
Contents
struct ibv_comp_channel *ibv_create_comp_channel(struct ibv_context *context); |
Description
ibv_create_comp_channel() creates a Completion event channel for an RDMA device context.
This Completion event channel is an abstraction introduced by libibverbs that does not exist in the InfiniBand Architecture verbs specification or RDMA Protocol Verbs Specification. A Completion event channel is essentially file descriptor that is used to deliver Work Completion notifications to a userspace process. When a Work Completion event is generated for a Completion Queue (CQ), the event is delivered via the completion event channel attached to that CQ. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.
One or more Completion Queues can be associated with the same Completion event channel.
Parameters
| Name | Direction | Description |
|---|---|---|
| context | in | RDMA device context that was returned from ibv_open_device() |
Return Values
| Value | Description | ||||
|---|---|---|---|---|---|
| Completion event channel | A pointer to the newly allocated Completion event channel | ||||
| NULL | On failure, errno indicates the failure reason:
|
Examples
Create a Completion event channel and destroy it:
struct ibv_comp_channel *event_channel; event_channel = ibv_create_comp_channel(context); if (!event_channel) { fprintf(stderr, "Error, ibv_create_comp_channel() failed\n"); return -1; } if (ibv_destroy_comp_channel(event_channel)) { fprintf(stderr, "Error, ibv_destroy_comp_channel() failed\n"); return -1; } |
FAQs
Why is a Completion event channel good for anyway?
Completion event channel is an object that helps handling Work Completions in a userspace process using event mode rather than polling mode.
Why would I want to read Work Completions in event mode?
Reading Work Completions in event mode is useful if your process doesn't need low latency or if you need to decrease the CPU usage; your process will go to sleep and will be awakened by the kernel when a new Work Completion will be available in a CQ, which is associated with that Completion event channel.
Can more than one CQ be associated with the same Completion event channel?
Yes. More than one CQ can be associated with the same Completion event channel. One can use several Completion event channels in order to "aggregate" different CQs. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.
How can I work with multiple Completion event channels in order to give different priority to different CQs?
You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group.
Share Our Posts
Share this post through social bookmarks.
Comments
Tell us what do you think.
Can you please explain what you mean by "You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group."
Thanks,
Hi.
I'll try to explain what I meant here:
Let's assume that you have several CQs: CQx1..CQxN and CQy (CQy has higher priority),
and your application needs to consume as less as CPU as possible
(i.e. work with events and not with polling when reading Work Completions).
Instead of using one Completion Event channel, you can have two of them,
one for the CQx QPs and one for the CQy, each of them will be handled in a different thread.
This way, as soon as CQy has an event, it will be handled
(i.e. it gets higher priority than the other CQs).
If your RDMA device supports multiple Completion vectors,
CQy may get a different Completion vector than the other,
and by doing this, get a better performance.
Instead of using one CQy, you can have CQy1..CQyM but the idea is the same.
I hope that I was clear with this explanation.
Thanks
Dotan
Dear Dotan
I want to associate different completion channel to each CQ i create. I have different threads waiting for notification from the CQ assigned to them. I think if i associate the same completion channel to all my CQ's and messages arrive at all of them simultaneously, i get a big performance hit. Should this happen? Can you share some code snippet where you are creating a new completion channel for each CQ. The code i use is
"cb->channel = ibv_create_comp_channel(cm_id->verbs);
cb->cq[SEND_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);
cb->cq[RECV_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);"
Since the context is the same, for each queue I suppose even if i call ibv_create_comp_channel() for each cq, they are shared. Also i am using the same channel for send/receive CQ. Will this cause a performance drop (due to some internal Locking mechanism).
I hope i have made my point clear. To summarize: I want all my CQ's to receive notification from the completion channel independent of each other. I have a large number of CQ's.
Warm regards
Omar Khan
Hi Omar.
I must admit that I never did it and never checked the performance difference.
But you need to create Completion channels and create every group of CQs with different
Completion channels.
IMHO, using multiple completion vectors may provide even better performance improvement than using the same completion vector in all the CQs.
Thanks
Dotan
Dear dotan
How do you know that your device supports multiple completion vectors? and what does it mean multiple completion vectors? if my device does not support multiple completion vectors, how can i set up a communication channel between multiple processes and have them communicate independently. can i have multiple threads using independent completion channels to notify completions from different processes?
Warm regards
Omar
Hi Omar.
struct ibv_context contains an attribute called: num_comp_vectors.
This value specify the number of completion vectors which the RDMA device supports.
I'm not consider my self an expert in this, but I can try to explain *my* rational why using multiple completion vectors improves the performance:
Using several completion vectors means that the interrupts will be spread across several vectors (which means that several cores in your system will handle them) and not only one vector.
I'm sure that you can find information in the internet about this..
If your device supports only one completion vector, I would try to set the affinity of the processes, that each one will use different core.
I hope that this answer helped you.
Thanks
Dotan
Hi Dotan,
Can I use the same event channel for both accepting connection and creating connection? In this way, a program can connect to itself, and the server-side event such as RDMA_CM_EVENT_CONNECT_REQUEST and client-side event such as RDMA_CM_EVENT_ADDR_RESOLVED can be polled in the same loop. It is possible?
Thanks,
Tingyu
Hi Tingyu.
The event channel is a mechanism for handle the events;
it doesn't have any notion of the side it serves.
For my understanding, the answer for you question is "yes".
It should be possible to use the same event channel for both accepting and initiating connections.
Thanks
Dotan
Hi Dotan,
Could multiple threads share one completion event channel associated with multiple CQs?
Thanks,
Lingyan
Yes.
Multiple threads can share one completion event channel associated with multiple CQs,
it will be hard to predict though which thread will get the event.
Thanks
Dotan
Hi Dotan,
I saw your reply above
https://www.rdmamojo.com/2012/10/19/ibv_create_comp_channel/?replytocom=701#reply
in your reply, you said "But you need to create Completion channels and create every group of CQs with different Completion channels."
What does every group of CQs mean?
As my realize, a completion channel is better to match to each CQ, or it is hard to know which CQ will receive event.
Do I miss something?
thanks
BR,
HuaiEn
Hi.
"Every group of CQs" means that if for example you create 100 CQs and you want to use 2 completion channels,
you can attach the first 50 CQs to the first completion channel and the last 50 CQs to the second completion channel.
When you get a CQ event, you get the CQ handle - so it is easy to know which CQ received the event.
However, depend on your flow, maybe you won't be able to predict ahead which CQ will get the event.
Is it more clear now?
Thanks
Dotan
Hi Dotan,
Thanks for clearly reply.
There is another question.
When do we need multiple cq?
Thanks a lot
Hi.
There may be several reasons for using multiple CQs, here are some reasons that I just thought about:
* Using *many* QPs and the number of generated completions may cause a CQ overrun to one CQ
* If you want to provide different QoS (application-wise) to different QPs
* If you want to provide different functionality to different QPs
* Separating Send and Receive Queues
* more?
It is up to the application to decide this, it isn't a technology decision ..
Thanks
Dotan