struct ibv_qp *ibv_create_qp(struct ibv_pd *pd,
                             struct ibv_qp_init_attr *qp_init_attr);

Description

ibv_create_qp() creates a Queue Pair (QP) associated with a Protection Domain.

The user can define the minimum attributes to the QP: number of Work Requests and number of scatter/gather entries per Work Request to Send and Receive queues. The actual attributes can be equal or higher than those values.

The struct ibv_qp_init_attr describes the requested attributes of the newly created QP.

struct ibv_qp_init_attr {
	void		       *qp_context;
	struct ibv_cq	       *send_cq;
	struct ibv_cq	       *recv_cq;
	struct ibv_srq	       *srq;
	struct ibv_qp_cap	cap;
	enum ibv_qp_type	qp_type;
	int			sq_sig_all;
};

Here is the full description of struct ibv_qp_init_attr:

The InfiniBand spec defines the QP transport type: Reliable Datagram. However, the RDMA software stack doesn't support it nor any RDMA device.

send_cq and recv_cq can be the same CQ or different CQs.
An RC and UD QPs always can be associated with an SRQ. There are RDMA devices which allow a UC QP to be associated with an SRQ as well. However, currently there isn't any indication to know that the RDMA device supports this.
struct ibv_qp_cap describes the size of the Queue Pair (for both Send and Receive Queues).

struct ibv_qp_cap {
	uint32_t		max_send_wr;
	uint32_t		max_recv_wr;
	uint32_t		max_send_sge;
	uint32_t		max_recv_sge;
	uint32_t		max_inline_data;
};

Here is the full description of struct ibv_qp_cap:

Sending inline'd data is an implementation extension that isn't defined in any RDMA specification: it allows send the data itself in the Work Request (instead the scatter/gather entries) that is posted to the RDMA device. The memory that holds this message doesn't have to be registered. There isn't any verb that specifies the maximum message size that can be sent inline'd in a QP. Some of the RDMA devices support it. In some RDMA devices, creating a QP with will set the value of max_inline_data to the size of messages that can be sent using the requested number of scatter/gather elements of the Send Queue. If others, one should specify explicitly the message size to be sent inline before the creation of a QP. for those devices, it is advised to try to create the QP with the required message size and continue decreasing it if the QP creation fails.

Parameters

Return Values

Examples

1) Create a QP with both CQ in the Send and Receive Queues and destroy it:

struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_qp *qp;
struct ibv_qp_init_attr qp_init_attr;
 
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
 
qp_init_attr.send_cq = cq;
qp_init_attr.recv_cq = cq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr  = 2;
qp_init_attr.cap.max_recv_wr  = 2;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
 
qp = ibv_create_qp(pd, &qp_init_attr);
if (!qp) {
	fprintf(stderr, "Error, ibv_create_qp() failed\n");
	return -1;
}
 
if (ibv_destroy_qp(qp)) {
	fprintf(stderr, "Error, ibv_destroy_qp() failed\n");
	return -1;
}

2) Create a QP with different CQs in the Send and Receive Queues:

struct ibv_pd *pd;
struct ibv_cq *send_cq;
struct ibv_cq *recv_cq;
struct ibv_qp *qp;
struct ibv_qp_init_attr qp_init_attr;
 
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
 
qp_init_attr.send_cq = send_cq;
qp_init_attr.recv_cq = recv_cq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr  = 2;
qp_init_attr.cap.max_recv_wr  = 2;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
 
qp = ibv_create_qp(pd, &qp_init_attr);
if (!qp) {
	fprintf(stderr, "Error, ibv_create_qp() failed\n");
	return -1;
}

3) Create a QP, which is associated with an SRQ:

struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_srq *srq;
struct ibv_qp *qp;
struct ibv_qp_init_attr qp_init_attr;
 
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
 
qp_init_attr.send_cq = send_cq;
qp_init_attr.recv_cq = recv_cq;
qp_init_attr.srq     = srq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr  = 2;
qp_init_attr.cap.max_send_sge = 1;
 
qp = ibv_create_qp(pd, &qp_init_attr);
if (!qp) {
	fprintf(stderr, "Error, ibv_create_qp() failed\n");
	return -1;
}

4) Create a QP with supports inline message:

struct ibv_pd *pd;
struct ibv_cq *cq;
struct ibv_qp *qp;
struct ibv_qp_init_attr qp_init_attr;
 
memset(&qp_init_attr, 0, sizeof(qp_init_attr));
 
qp_init_attr.send_cq = cq;
qp_init_attr.recv_cq = cq;
qp_init_attr.qp_type = IBV_QPT_RC;
qp_init_attr.cap.max_send_wr  = 2;
qp_init_attr.cap.max_recv_wr  = 2;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
qp_init_attr.cap.max_inline_data = 512;
 
qp = ibv_create_qp(pd, &qp_init_attr);
if (!qp) {
	fprintf(stderr, "Error, ibv_create_qp() failed\n");
	return -1;
}

FAQs

Why is a QP good for anyway?

QP is the actual object that sends and receives data in the RDMA architecture (something like a socket).

Are socket and QP equivalent?

Not exactly. A socket is an abstraction, which is maintained by the network stack and doesn't have a physical resource behind it. A QP is a resource of an RDMA device and a QP number can be used by one process at the same time (similar to a socket that is associated with a specific TCP or UDP port number)

Can I associate several QPs with the same SRQ?

Yes. you can.

Which QP Transport Types can be associated with an SRQ?

RC and UD QPs, can be associated with an SRQ by all RDMA devices. In some RDMA devices, you can associate a UC QP with an SRQ as well.

Do I need to set the Receive Queue attributes if I associate a QP with an SRQ?

No, you don't have to do it. The Receive Queue attributes are completely ignored if the QP is being associated with an SRQ.

Can I use the same CQ in both Send in Receive Queues?

Yes. you can.

Can I use one CQ in the Send Queue and another CQ in the Receive Queue?

Yes. you can.

How can I know what is the maximum message size that can be sent inline in a QP?

You can't know this information. This information is unavailable. You find this information by trial and error.

I created a QP with transport type X and the QP was created successfully. I tried to create a QP with transport type Y and the QP creation failed. What happened?

The value in dev_cap.max_sge and dev_cap.max_qp_wr reports the supported values of scatter/gather entries and Work Requests that are supported by any QP's transport type. However, for a specific RDMA device, there may be QP's transport types that cannot be created with those maximum values. Using trial and error, one should get the right attributes for this specific RDMA device.

The device capabilities reported that max_qp_wr/max_sge is X, but when I tried to create a QP with those attributes it failed. What happened?

The value in dev_cap.max_sge and dev_cap.max_qp_wr reports the maximum supported values of scatter/gather entries and Work Requests that are supported by any Work Queue (Send and Receive). However, for a specific RDMA device, there may be other considerations for the Send or Receive Queue that prevent a QP to be created with those maximum values. Using trial and error, one should get the right attributes for this specific RDMA device.

Written by: Dotan Barak on December 21, 2012.on June 26, 2015.