How to flush a Work Queue?
When one wishes to stop the outstanding Work Requests from being processed, flushing the Work Queues may be useful. The most common reason for doing this is to reclaim the memory buffers that the Work Requests refer to. Another reason is to stop working with specific Work Queues (for example, stop sending/receiving data by a specific local QP).
Once Work Requests were posted to a Work Queue, one cannot know what is their status (until the Work Completions for them were polled):
- Maybe for some of them, the processing hasn't started
- Maybe for some of them, the processing has started, but wasn't completed
- Maybe for some of them, the processing was completed, but the Work Completion for them weren't polled yet
Flushing a Work Queue will only take care of Work Requests that weren't completed. Only Work Queue(s) can be flushed and not a specific Work Requests.
In this post, we will describe how to flush the Work Requests in the Send and in the Receive Queues.
Flushing a Send Queue
The Work Requests in the Send Queue initiate data transfer either from local to remote memory or backwards (depends on the used opcode).
When one flushes the Send Queue, he doesn't know what is the status of the outstanding Work Requests in it and he should be ready for the case that part (or all) of them were completed.
In order to flush the Send Queue, one should call ibv_modify_qp() and move the QP to the Error state.
This action will flush all of the Work Requests in the Send Queue that weren't completed and for each of them a Work Completion with the "Flushed in Error" status will be generated.
The Send Queue may be flushed if the QP will be transferred by the RDMA device to the Error state (in RC QP) or to the Send Queue Error state (in UC and UD QPs) because of an error in the Send Queue.
Flushing a Receive Queue of a QP which isn't associated with an SRQ
The Work Requests in the Receive Queue specify to which memory buffers the incoming messages to this QP, that consume a Receive Request, will be written.
When one flushes the Receive Queue, he doesn't know what is the status of the outstanding Work Requests in it and he should be ready for the case that part (or all) of them were completed.
In order to flush a Receive Queue in a QP that isn't associated with an SRQ, one should call ibv_modify_qp() and move the QP to the Error state.
This action will flush all of the Work Requests in the Receive Queue that weren't completed and for each of them a Work Completion with the "Flushed in Error" status will be generated.
The Receive Queue may be flushed if the QP, from any transport type, will be transferred by the RDMA device to the Error state.
Flushing a Receive Queue of a QP which is associated with an SRQ
The Work Requests in the Receive Queue specify to which memory buffers the incoming messages to any QP, which is associated with the SRQ, that consume a Receive Request will be written.
When one flushes the Receive Queue, he doesn't know what is the status of the outstanding Work Requests in it and he should be ready for the case that part (or all) of them were completed.
In order to flush the Receive Queue in a QP that is associated with an SRQ, one should call ibv_modify_qp() and move the QP to the Error state.
This action will flush all of the Work Requests in the QP's Receive Queue that were fetched from the SRQ and their processing wasn't completed. For each of them a Work Completion with the "Flushed in Error" status will be generated.
The Receive Queue may be flushed if the QP, from any transport type, will be transferred by the RDMA device to the Error state.
The asynchronous event IBV_EVENT_QP_LAST_WQE_REACHED indicated that this flush is completed, and there are no more WQEs on Receive Queue of that QP.
The SRQ itself cannot be flushed and the posted Receive Requests in it cannot be reclaimed. One possible workaround for this can be to associate a QP with this SRQ and consume all of the WRs from this SRQ by sending messages with an opcode that consumes a Receive Request.
Share Our Posts
Share this post through social bookmarks.
Comments
Tell us what do you think.
Hello,
I need to flush all Work Request posted to a QP then reset the QP and re-initialize it again through INIT, RTR, RTS states.
Should I wait for all Work Requests to complete with the error state, after the QP has been transferred to the Error state before moving the QP to the RST state ?
If I will move a QP to the RST, INIT, RTR, RTS states immediately after the QP was transferred to the Error state, will the QP be flushed as well ?
Hi Gregory.
Moving a QP to the Reset state will flush all the outstanding WRs and remove them from both
Work Queues and Completion Queues.
If you don't need those flushed Work Requests (since they don't contain any critical information for you),
you can just move the QP to the Reset state and then to INIT, RTR, RTS state and this will be o.k.
Thanks
Dotan
Hi Dotan,
Thank you for the wonderful posts.
I had a query regarding the completion of the flush event.
I have the conn. event handler as a separate thread.
The poll thread for completion queue is also a seperate thread.
The main flow of the application a third thread.
So, when a connection related event is issued like rdma_connect on the main thread and then it sleeps. The handler thread once the application level handling has been done wakes the main thread up.
Now, once I do a listener disconnect or if the remote side disconnects or terminates, I find that the work request queues(send/receive) are flushed to provide the status as work completion in form of a FLUSH error. Perfect.
I have couple of queries.
1. How do I know if all the messages have been flushed. Will there be some event generated for it?
2. I do a rdma_disconnect on a listener and sleeps. Then the connection disconnect event handler wakes up the main thread thats sleeping. I, then, go on to destroy the QPs associated with the listener on the main thread. I find that the flushing of incomplete messages stops prematurely. Is there a way I can get around this?
Thanks,
George.
Answers:
1) No. There won't be any special event that specify that all messages have been flushed;
you simply need to count the Work Completions and figure it up by yourself.
2) Flushing of incomplete messages will be stopped if the QP state moved to Reset,
or the QP is destroyed. Maybe this is the case for you.
I would suggest to move the QP to error, wait until all messages are flushed,
and only then destroy the QP.
Thanks
Dotan
Hi Dotan,
When will a QP enter error state if we do not manually set it. The queue pair starts flushing the messages before I call rdma_disconnect and when the other server has gone down. Is it the case that when the connection is severed at the remote end, then too flushing can start without rdma_disconnect being called explicitly.
Thanks,
George.
Hi George.
The QP will be transitioned automatically to the Error state in the following scenarios:
In the local side:
* If the Receive Queue got an error
* If the Send Queue got an error and the QP transport type is Reliable
In the receive side:
* If the remote QP got an error and the QP transport type is Reliable
I hope that this helped you.
Thanks
Dotan
Hi...
I am getting ENOMEM error while ib_post_recv function calling.
May I get help which can solve this problem?
Hi.
I would suggest you to check if the QP.RQ is full or the number of scatter entries in the Receive Request exceeds the QP capabilities.
Thanks
Dotan
If the QP using a srq, how could I know when "all messages are flushed"?
Hi.
I added the following line to this post, I hope that this will answer your question:
"The asynchronous event IBV_EVENT_QP_LAST_WQE_REACHED indicated that this flush is completed, and there are no more WQEs on Receive Queue of that QP."
Thanks
Dotan