In the last post we discuss the various QP states. In this post I will try to explain why those states are needed and how to use the QP in each state.

Reset state

A QP can get to the Reset by two ways:

1. By creating it
2. By moving it to Reset state from any other state by calling  ibv_modify_qp()

When a QP is being created, it takes some time to create it (there are context switches, memory allocations for the work queue buffers, QP number allocation, etc.).

If one needs to use a new QP in the fast path, a better replacement for the following:

• Create a QP
• Use the QP (modify to RTS and send/receive data)
• Destroy the QP

Can be: create a QP only once and then do the following, when needed:

• Use the QP (modify to RTS and send/receive data)
• Modify the QP to Reset for later use

when the QP won't be needed anymore, it can be destroyed.

Reusing the same QP may be a faster solution than creating it each time. Moving a QP from any other state to the Reset state cleans all the "leftovers" from the previous transactions (such as Work Requests and completions in the send and the receive queues).

However, using the same QPs between the same nodes is discouraged since unexpected results may occur (packet lost).

Moving the QP to the Reset state will prevent the QP from sending or receiving packets.

Init state

When moving the QP to the RTR state, the QP will handle incoming packets. If for any reason (for example: OS scheduler didn't give time to that process) after the transition to the RTR state, Work Requests weren't posted to the receive queue of that QP, Receiver Not Ready (RNR) error may occur to the requester of those packets. In order to prevent this from happening, one can post receive requests to the QP when it is in the Init state. Those Work Request won't be processed until the QP will be transitioned to the RTR state.

RTR state

In this state, the QP handles incoming packets. If the QP don't be used as a requester (i.e. Work Requests won't be posted to its send queue), the QP may stay in the RTR state.

RTS state

In most of the applications, the QPs will be transitioned to the RTS state. In this state, the QP can send packets as a requester and handle incoming packets.

Even if your QP won't act as a requester, it can be transitioned to this state.

SQD state

If one wishes to change some of the send queue attributes, he should transition the QP to the SQD state, modify the needed attributes and then transition the QP back to the RTS state to be able to continue sending packets.

SQE state

A QP can be transitioned to this state only automatically by the device in case of a Work Request, in the send queue, that completed with error. In this state, the send queue can't send packets as a requester. If one wishes to recover the QP from this state, he should modify the QP back to the RTS state.

Error state

QP can be moved to the Reset by two ways:

1. Automatically, by the RDMA device in case of a completion with error
2. Move the QP to Error state from any other state

If there was an error, the first completion in the Completion Queue of the Queue (Send or Receive Queue) that got the error will hold a status that indicates the reason of the error. The rest of the Work Requests in that Queue and in the other queue will be flushed in error.

This state is useful for reclaiming all the Work Requests back to free their buffers (for example, if the attribute wr_id in the Work Requests specifies the buffers that were used by this Work Request).

Moving the QP to the Error state will prevent the QP from sending or receiving packets.

Destroying the QP

This isn't a QP state per se, but when a user finished working with a QP and he wishes to prevent the QP from sending or receiving packets, he can destroy it.

Summary

In this post, we discussed how to use the QP in each state.

Written by: Dotan Barak on May 10, 2012.on February 16, 2015.