Sometimes, within an InfiniBand fabric (i.e. a set of endnodes physically connected by cables to switches and routers) there is a need to provide an isolation between specific nodes and create "virtual fabrics" that only they will be able to communicate. In Ethernet, this is called a Virtual LAN (VLAN) and it defines a broadcast domain. However, in InfiniBand broadcast isn't supported, so it just provides virtual isolation. This is called Partitioning. And it is a little bit more powerful than VLAN.

Partitioning is being represented (and implemented) using Partition Key (P_Key) which is a 16-bits value that has two parts:

• The msb is the membership bit:
  • 0: limited membership
  • 1: full membership
• The lower 15 bits are the key portion which defines the partition value

Only QPs that are configured to be in the same partition can communicate and it least one of the QPs should be a full member in the partition. We'll discuss this in details below.

Partitioning provides isolation between an InfiniBand fabric and is not related to the boundaries of subnets.

P_Key table

P_Key Table is a table of P_Keys that exists in each endnode Port. This table size is an attribute of the port and it can be found in port_attr.pkey_tbl_len. This table is configured by the Partition Manager (PM), which is usually part of the Subnet Manager (SM). This table has at least one valid P_Key (a P_Key that its key portion isn't zero) which is the default P_Key 0xFFFF.

The P_Key table may contain multiple values, i.e. be a member of multiple partitions, in some of them it may be a full member and in some of them it may be a partial member. It even can be full and partial member in the same partition.

Every QP is configured with an index to the P_Key table. Using ibv_modify_qp(), which is called during the Connection Establishment, one can set an index in the P_Key table of the port that this the QP is associated with.

The values in the P_Key table aren't constant and they may change, so the application must be ready for this to happen.

P_Key in the Send Queue

Posting a SR to a QP initiate packet(s) to be sent. Every packet carries a P_Key value. When sending a packet, it will care the P_Key that was configured to be used by the QP.

P_Key in the Receive Queue

The P_Key value of any incoming packet will be compared with the P_Key that was configured in the receiver QP. The following tests will be performed:

• Check that the key portion of both P_Keys are equal, i.e. both of them are members in the same partition.
• Check that at least one of the P_Key values are full members.

If at least one of the above tests will fail, this will be considered P_Key violation and the packet will be silently dropped, the port_attr.bad_pkey_cntr will be increased if supported by the device (IBV_DEVICE_BAD_PKEY_CNTR is set in dev_cap.device_cap_flags). The receiver QP state won’t be changed.

The important things are the P_Key values and not their location within the P_Key table. A specific P_Key value may be found in different ports in the subnet in different indexes.

From the above tests, one can learn that Limited members cannot accept traffic from other Limited members, and communication is allowed between every other combination of membership types, i.e. at least one of the sides has to be a full member.

Let's give an example:

• QPA is associated with the P_Key value 0x8001
• QPB is associated with the P_Key value 0x0001
• QPC is associated with the P_Key value 0x0001
• QPD is associated with the P_Key value 0x8002

QPA and QPB can communicate (since both of them are members in the Partition 0x0001 and QPA is a full member).

QPA and QPC can communicate (since both of them are members in the Partition 0x0001 and QPA is a full member).

QPB and QPC cannot communicate (since both of them are members in the Partition 0x0001 but none of them is a full member).

QPD and any other QP in our example cannot communicate (since they aren't members of the same partition).

P_Key values

The default P_Key is a special value (0xFFFF) providing Full membership in the default partition.

Invalid Partition Key: is a special value that indicates that the Partition Key Table entry does not contain a valid key.

Written by: Dotan Barak on April 30, 2014.on October 13, 2018.