I'd appreciate if you could shed some light on the following subject.
As per "Infinibeand Network Architecture"(* see quotations below), during RDMA Read request, the responder's RQ logic is allowed to re-read from memory when re-trying. Can we totally disable this functionality by setting retry_cnt=rnr_cnt=0?
In other words, can one be sure in such a case, that a single RDMA Read request will never cause double read of any remote location? (This can be crucial if it's a device memory, and an access might have side effects.)

Thanks!

* Infiniband Network Architecture, p.34:
<<
This service is referred to as "reliable" for the following reasons:
-- The destination QP's RQ Logic verifies the PSN in each request packet to ensure that all of the message's request packets are received in order, that none are missing, and, if any duplicate request packets are received, that they are only processed once. There is one exception; upon receipt of a duplicate memory read, the data is read from memory again.

>>

And in details (p.394):
<<
Effects of Retry on Requester and Responder The following list defines the sequence of events:

6. RQ Logic receives duplicate copy of request. When the second copy (the retried one) of the request packet eventually arrives at the RQ Logic, it has the same PSN as the original request packet. Its PSN, therefore, falls into the Ack’ d PSN region (also referred to as the Duplicate region), thereby identifying it as a duplicate request packet. The actions taken by the RQ Logic are determined by the request type:

— If it’ s a duplicate RDMA Read:
– The RQ Logic re-executes the read from its local memory.
– It sends back the requested read data in a series of one or more RDMA Read response packets.
– The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
– The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
– After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN.

Hi, if I make a connected qp to error state, is it possible that peer side can detect this.

For example, if we use tcp socket to communicate, if we close one side's socket, other side will get "0" when read. So peer side can know peer socket already closed. Is it possible that rdma can achieve this?

Thanks!

Hi Dotan,
the code is

attr.qp_state = IBV_QPS_RTR;
attr.path_mtu = mtu;
attr.dest_qp_num = rmsg.qp_num[0];
attr.rq_psn = rmsg.psn[0];
attr.ah_attr.dlid = rmsg.lid;
attr.max_dest_rd_atomic = 1;
attr.min_rnr_timer = 12;
attr.ah_attr.is_global = 0;
attr.ah_attr.sl = 0;
attr.ah_attr.src_path_bits = 0;

if (ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |
IBV_QP_DEST_QPN | IBV_QP_RQ_PSN | IBV_QP_MIN_RNR_TIMER |
IBV_QP_MAX_DEST_RD_ATOMIC)){
cerr << "Failed to modify QP 1 to RTR." << endl;
abort();
}

lid, qp_num and psn are sent via TCP sockets.

Thanks

Hi Dotan：
After reading this blog，I feel confused about the required and optional attributes of QP on status transitions 。 the Linux manpages 、rdma_aware_network_programming_user_manual and your blog have different description 。which is right? Thanks

Hi Dotan:
These days I encounter an error:after run ibv_post_send to issue an RDMA READ operation, there is nothing happened, this means that the RDMA READ operation don't appears in CQ. After to read "Help, I've posted and Send Request and it wasn't completed with a corresponding Work Completion. What happened?" in "ibv_post_send" page, I decide to try advise in it. The problem is I can't modify qp attribute, the ibv_modity_qp return -22 all the time, below is my code:
int ret = 0;
rdma_addrinfo* addrinfo;
rdma_addrinfo hints = {};
struct ibv_qp_attr qp_attr;
hints.ai_port_space = RDMA_PS_TCP;
struct ibv_qp *client_qp = nullptr;
if (rdma_getaddrinfo(const_cast(host.c_str()),
const_cast(port.c_str()), &hints, &addrinfo)) {
return errors::InvalidArgument(
strerror(errno), ": ", "cannot connect to rdma://", host, ":", port);
}

ibv_qp_init_attr init_attr = {};
init_attr.qp_type = IBV_QPT_RC;
init_attr.cap.max_recv_wr = 1;
init_attr.cap.max_send_wr = 1024;
init_attr.cap.max_recv_sge = 1;
init_attr.cap.max_send_sge = 1;

rdma_cm_id* id;
if (rdma_create_ep(&id, addrinfo, nullptr, &init_attr)) {
rdma_freeaddrinfo(addrinfo);
return errors::Unavailable(strerror(errno), ": ",
"cannot create endpoint to rdma://", host, ":",
port);
}
rdma_freeaddrinfo(addrinfo);

client_qp = id->qp;

// motify qp's attr.
bzero(&qp_attr, sizeof(qp_attr));
qp_attr.qp_state = IBV_QPS_RTS;
qp_attr.timeout = 14;
qp_attr.retry_cnt = 7;

ret = ibv_modify_qp(client_qp, &qp_attr,
IBV_QP_TIMEOUT |
IBV_QP_STATE |
IBV_QP_RETRY_CNT);
if (ret) {
fprintf(stderr, "failed to motify qp, %s\n", strerror(errno));
abort();
}
if (rdma_connect(id, nullptr)) {
rdma_destroy_ep(id);
return errors::Unavailable(strerror(errno), ": ",
"cannot connect to rdma://", host, ":", port);
}

My rdma NIC type is RoCE, could you help me to see why I can't motify qp statue?