ibv_create_ah()
Contents
struct ibv_ah *ibv_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr); |
Description
ibv_create_ah() creates an Address Handle (AH) associated with a Protection Domain.
This AH will later be used when a Send Request (SR) will be posted to a Unreliable Datagram QP.
The struct ibv_ah_attr describes the requested attributes of the newly created AH.
struct ibv_ah_attr { struct ibv_global_route grh; uint16_t dlid; uint8_t sl; uint8_t src_path_bits; uint8_t static_rate; uint8_t is_global; uint8_t port_num; }; |
Here is the full description of struct ibv_ah_attr:
grh | Attributes of the Global Routing Headers (GRH), as described in the table below. This is useful when sending packets to another subnet or to a multicast group |
dlid | If the destination is on the same subnet, the LID of the port to which the subnet delivers the packets to. If the destination is on another subnet, the LID of the Router |
sl | 4 bits. The Service Level to be used |
src_path_bits | The used Source Path Bits. This is useful when LMC is used in the port, i.e. each port covers a range of LIDs. The packets are being sent with the port's base LID, bitwised ORed with the value of the source path bits. The value 0 indicates the port's base LID is used |
static_rate | A value which limits the rate of packets that being sent to the subnet. This can be useful if the rate of the packet origin is higher than the rate of the destination |
is_global | If this value contains any value other than zero, then GRH information exists in this AH, thus the field grh if valid |
port_num | The local physical port that the packets will be sent from |
struct ibv_global_route describes the values to be used in the GRH of the packets that will be sent when using this AH.
struct ibv_global_route { union ibv_gid dgid; uint32_t flow_label; uint8_t sgid_index; uint8_t hop_limit; uint8_t traffic_class; }; |
Here is the full description of struct ibv_global_route:
dgid | The GID that is used to identify the destination port of the packets |
flow_label | 20 bits. If this value is set to a non-zero value, it gives a hint for switches and routers with multiple outbound paths that these sequences of packets must be delivered in order, those staying on the same path, so that they won't be reordered. |
sgid_index | An index in the port's GID table that will be used to identify the originator of the packet |
hop_limit | The number of hops (i.e. the number of routers) that the packet is permitted to take before being discarded. This ensures that a packet will not loop indefinitely between routers if a routing loop occur. Each router decrement by one this value at the packet and when this value reaches 0, this packet is discarded. Setting the value to 0 or 1 will ensure that the packet won't leave the local subnet. |
traffic_class | Using this value, the originator of the packets specifies the required delivery priority for handling them by the routers |
Parameters
Name | Direction | Description |
---|---|---|
pd | in | Protection Domain that was returned from ibv_alloc_pd() |
attr | in | Requested attributes for the Address Handle |
Return Values
Value | Description | ||||
---|---|---|---|---|---|
AH | pointer to the newly allocated Address Handle | ||||
NULL | On failure, errno indicates the failure reason:
|
Examples
Create an Address Handle and destroy it (this example assumed that the variables dlid, sl and port are declared and initialized with valid values):
struct ibv_pd *pd; struct ibv_ah *ah; struct ibv_ah_attr ah_attr; memset(&ah_attr, 0, sizeof(ah_attr)); ah_attr.is_global = 0; ah_attr.dlid = dlid; ah_attr.sl = sl; ah_attr.src_path_bits = 0; ah_attr.port_num = port; ah = ibv_create_ah(pd, &ah_attr); if (!ah) { fprintf(stderr, "Error, ibv_create_ah() failed\n"); return -1; } if (ibv_destroy_ah(ah)) { fprintf(stderr, "Error, ibv_destroy_ah() failed\n"); return -1; } |
FAQs
Why is an AH good for anyway?
AH is used when calling ibv_post_send() to an UD QP. This object describes the path from the requester side to the responder side.
Can I use one AH with different QPs?
Yes. This can be done as long as all of those QPs and the AH are associated with the same PD.
How can I get the needed information for the AH when calling ibv_create_ah()?
There are several ways to obtain this information:
- Perform path query to the Subnet Administrator (SA)
- Out of band connection to the remote node, for example: using socket
- Using well-known values, for example: this can be done in a static subnet (which all of the addresses are predefined) or using multicast groups
Comments
Tell us what do you think.
Can you describe how to leverage VLANs over RoCE? Assuming the VLAN ID is preset in the DLID somehow and the priority is explicitly provided via the SL on AH create. Any caveats with PFC?
Hi Omar.
First, I plan to cover all of the RDMA verbs since I don't think that there is another point of knowledge at this level (detailed description and code samples).
In the future, I plan to cover RoCE as well and explain how to port an application over IB to an application over RoCE.
Hi this is a great resource. Can you describe how this maps to VLAN IDs and priorities in RoCE? Assuming the VLAN ID was previously obtained somehow into the DLID and the SL is selected at AH create time. How does one discover/ensure the verb consumer selected VLAN and priorities are valid?
Hi Omar.
Thanks for the complements.
:)
As I wrote in my previous reply: first, I plan to cover all of the RDMA verbs since I don't think that there is another point of knowledge at this level (detailed description and code samples).
In the future, I plan to cover RoCE as well and explain how to port an application over IB to an application over RoCE.
Could you please clarify how to fill ah_attr.sl and may be some other fields in case of 3D Torus cluster topology.
ah_attr.sl = sl
Can sl value be used from ibv_query_port()?
Hi.
The ibv_query_port() will tell you the number of supported VLs by the port and the SL that you need to use to get to the SM/SA.
Did you try to send a path query to the SA (Subnet Administrator)?
I think that you would like to read the following article:
http://www.disca.upv.es/jflich/papers/icpp03.pdf
Thanks
Dotan
Dotan,
Thank you very much for response and the reference.
No, I did not try to send a path query to the SA. Currently I do not have access to 3D Torus clusters to make experiments. I implemented the module that uses Infiniband UD transport where I just set ah_attr.sl = 0 and customer asked the support for 3D Torus cluster. I investigated mvapich2.1 sources and did not find any special changes according sl for UD. For RC they have a special function like get_path_record().