ibv_query_device()
Contents
int ibv_query_device(struct ibv_context *context, struct ibv_device_attr *device_attr); |
Description
ibv_query_device() returns the attributes of an RDMA device that is associated with a context.
Here is the full description of struct ibv_device_attr:
Name | Description |
---|---|
fw_ver | A NULL terminated string which describes the Firmware version of the RDMA device |
node_guid | GUID (in network byte order) associated with the RDMA device. This is the same GUID that was returned by ibv_get_device_guid() |
sys_image_guid | GUID (in network byte order) associated with this RDMA device and other devices which are part of a single system. For example: multiple Switch chips in the same Core switch |
max_mr_size | Size (in bytes) of the largest contiguous memory block that can be registered by this device |
page_size_cap | Memory page sizes supported by this device |
vendor_id | IEEE device's Vendor |
vendor_part_id | Devic's Part ID, as supplied by the Vendor |
hw_ver | Devic's Hardware version, as supplied by the Vendor |
max_qp | Maximum number of QPs, of UD/UC/RC transport types, supported by this device |
max_qp_wr | Maximum number of outstanding work requests on any Send or Receive Queue supported by this device |
device_cap_flags | Device's supported capabilities as bitwise ORed of the following enumerated values:
|
max_sge | Maximum number of scatter/gather entries per Send or Receive Work Request, in a QP other than RD, supported by this device |
max_sge_rd | Maximum number of scatter/gather entries per Send or Receive Work Request, in an RD QP, supported by this device. If RD isn’t supported by this device, this value is zero |
max_cq | Maximum number of CQs supported by this device |
max_cqe | Maximum number of entries in each CQ supported by this device |
max_mr | Maximum number of MRs supported by this device |
max_pd | Maximum number of PDs supported by this device |
max_qp_rd_atom | Maximum number of RDMA Reads & atomic operations that can be outstanding per QP with this device as the target, if such are supported |
max_ee_rd_atom | Maximum number of RDMA Reads & atomic operations that can be outstanding per EEC with this device as the target, if such are supported |
max_res_rd_atom | Total maximum number of resources used for RDMA Reads & atomic operations by this device as the target, if such are supported |
max_qp_init_rd_atom | Maximum depth per QP for initiation of RDMA Read & atomic operations by this device, if such are supported |
max_ee_init_rd_atom | Maximum depth per EEC for initiation of RDMA Read & atomic operations by this device, if such are supported |
atomic_cap | Ability of the device to support atomic operations. The support level can be one of the following enumerated values:
|
max_ee | Maximum number of EE contexts supported by this device. If RD isn’t supported by this device, this value is zero |
max_rdd | Maximum number of RDDs supported by this device. If RD isn’t supported by this device, this value is zero |
max_mw | Maximum number of MWs supported by this device. If MWs aren't supported by this device, this value is zero |
max_raw_ipv6_qp | Maximum number of Raw IPv6 Datagram QPs supported by this device. If Raw IPv6 Datagram QPs aren't supported by this device, this value is zero |
max_raw_ethy_qp | Maximum number of Raw Ethertype Datagram QPs supported by this device. If Raw Ethertype Datagram QPs aren't supported by this device, this value is zero |
max_mcast_grp | Maximum number of multicast groups supported by this device. If unreliable multicast isn't supported by this device, this value is zero |
max_mcast_qp_attach | Maximum number of QPs per multicast group supported by this device. If unreliable multicast isn't supported by this device, this value is zero |
max_total_mcast_qp_attach | Maximum number of total QPs which can be attached to a multicast groups for this device. If unreliable multicast isn't supported by this device, this value is zero |
max_ah | Maximum number of AHs supported by this device |
max_fmr | Maximum number of FMRs supported by this device. If FMRs aren't supported by this device, this value is zero |
max_map_per_fmr | Maximum number of maps per FMR supported by this device. If FMRs aren't supported by this device, this value is zero |
max_srq | Maximum number of SRQs supported by this device. If SRQs aren't supported by this device, this value is zero |
max_srq_wr | Maximum number of outstanding work requests in an SRQ supported by this device |
max_srq_sge | Maximum number of scatter entries per Receive Work Request, in an SRQ, supported by this device |
max_pkeys | Maximum number of partitions supported by this device |
local_ca_ack_delay | Local CA ACK Delay. This value specifies the maximum expected time interval between the local device receiving a message and transmitting the associated ACK or NAK. |
phys_port_cnt | Number of physical ports on this device |
The RDMA device attributes, returned by ibv_query_device(), are constant and won't be changed by the device or by the SM, thus a program can call to this verb and save those attributes for later use.
The maximum values returned by ibv_query_device() are the upper limits of supported resources of the device. However, it may not be possible to use these maximum values since the actual number of any resource that can be created may be limited by the machine configuration, the amount of host memory, user permissions, and the amount of resources already in use by other users/processes.
Parameters
Name | Direction | Description |
---|---|---|
context | in | RDMA device context, that was returned from ibv_open_device() |
device_attr | out | RDMA device attributes |
Return Values
Value | Description |
---|---|
0 | On success |
errno | On failure |
ENOMEM | Out of memory |
Examples
Query a device attributes:
struct ibv_device_attr device_attr; int rc; rc = ibv_query_device(ctx, &device_attr); if (rc) { fprintf(stderr, "Error, failed to query the device '%s' attributes\n", ibv_get_device_name(device_list[i])); return -1; } |
FAQs
ibv_query_device() indicated that I can create X elements from a resource, but I was able to create only Y (where Y is less than X), is this is o.k.?
Yes, this may happen since the values reported by ibv_query_device() are the upper limits supported by the device. If any other processes/modules created elements from this resource too, the available number this resource was decreased.
Can I know how many elements of a resource are available for me to create?
No, you can't. This value depends on a lot of factors.
Can I know how many elements of a resource were created by other processes/modules?
No, you can't. Currently, the RDMA stack doesn't support this feature.
Comments
Tell us what do you think.
Hi Dotan,
I have a general question about adapters supporting RDMA over Ethernet. Does "Infiniband Spec" require vendors to support end to end flow control (based on
credits) or is it optional? OR is it the
verbs application developers duty to
figure out if receiver is ready and has
recv wqe posted before sending the data?
Hi Jay.
Sorry it took some time to answer.
Short answer:
HCAs are required to implements end to end credits. It is a good practice for
software to have flow control as well.
Long answer:
RC responders of HCAs shall support end to end credit generation (C9-158.2.1).
However, TCAs are not required to (it is not a must).
Some implementations weren't fully optimized for this case of running out of credits
and therefore many applications use software flow control on top of this.
The class of applications that use SW flow control is to ensure specific control message can always be sent to avoid cases of deadlock or forward progress, and hence needed to guarantee special room in remote machine.
Last point, is when SRQ is being used, also flow control is not reflected. And hence even in HCA, if the app is using SRQ, then credits will not be reflected on the wire. (in such a case it is even hard to do SW flow control).
I hope that this answer helps you
Dotan
Hi:
What is the abbreviation of "rd" in max_qp_rd_atom and "rdd" in max_rdd? and "ee" in max_ee? thanks!
Hi.
Depends where:
"max_sge_rd" -> RD = Reliable Datagram
"ee" -> End to End context (used by RD QP)
"max_qp_rd_atom" -> RD = Read
Usually, from the context, one can understand which one to use.
Thanks
Dotan
Is there a tool to show these attributes?
Hi.
ibv_devinfo will print those attributes.
Thanks
Dotan