I have a question about whether a context switch is occurred or not during an RDMA operation. Here (page 15) it is shown that a user space verbs call results in a call of the hardware specific driver (eg mlx4). That "lives" in kernel space. So, ibv_post_send() (RDMA mode) causes a context switch, or not? Can you clarify this for me please.

Also, if ibv_post_send() never causes a context switch, then why there is an implementation of ibv_post_send() in the linux kernel. When is this function (inside the kernel) called?

Thanks!

when i send a 1024 bite block by IBV_WR_RDMA_WRITE mode，everything is ok， but if block size is set larger (ex 4096 bite),I get a IBV_WC_LOC_PROT_ERR err and then many IBV_WC_WR_FLUSH_ERR err for send cq , can u help me

Hi Dotan, I'm running into a problem with ibv_post_send and hoping you can provide some guidance. I've adapted the rc_ping_pong program to exchange 312 byte messages among nodes in a 32-machine IB cluster, except that I use an epoll() based mechanism to call ibv_poll_cq(). Several messages later (around 58900 to be exact), ibv_post_send() fails returning ENOMEM and errno set to 2. Both sides of the connection are in good states: IBV_PORT_ACTIVE & IBV_QPS_RTS. When I keep track of sends posted vs sends completed I find that during the failure (posted-completed) = 31, always. However I have only max_send_wr=1 when I created the qp. So I'm not sure what's going on. On the receive side I guarantee posts (rx_depth=800 and whenever it drops to 400 I post 400 more). Any help is much appreciated, and if you need further clarifications please let me know.
Thanks much
Sara

Hi Dotan,

I'm receiving 'remote invalid request error' (IBV_WC_REM_INV_REQ_ERR) with RDMA_READ requests. I checked buffer sizes, access rights, and QP-type and all seams fine to me. RDMA_WRITE works and since the only difference is the opcode (as far as I know), I don't understand the issue.

BTW: I'm new to RDMA programming and your side really helps a lot!

Thanks so far.

Hello Dotan

in ibv_post_send:
1. Are the ibv_send_wr list, and its sg_list destroyed automatically when the operation completes.
2. Or can I destroy them after the method call returns.
3. They have to be kept alive till receiving work completion.

Boris.
Thanks.

Hi,
I'm new to RDMA and run into a weird behavior, which I was hoping you could clarify for me:

I'm using IBV_WR_SEND to send a struct-object which contains some information needed for an RDMA-read later on (rkeys, address and so on).
Now in principle this works fine, but the strange behavior is that only if the object-size is a multiple of 2, does it work correctly. So I tried these cases:
sizeof(message) -> 16. This works
sizeof(message) -> 24. The last object-attribute is always wrong, the rest is correct.
sizeof(message) -> 32. This works again.

Is this normal? I have only seen restrictions about the minimum/maximum message size, but nothing that would hint at an additional restriction of this kind. Or did I something wrong somewhere?

Thank you very much!
Martin

Hi Dotan,

I have a question about P_KEYS in BTH header. Once a relation is established between two QP's, both ends can modify the qp attribute pkey_index. Can both ends use different pkey_index (and ultimately different pkeys) ? i.e A can say B is using Pkey=X and B can say A is using Pkey=Y.
Thanks,

Jay

Hi Dotan,

I have a question about how WRs are finished. Suppose I have built a RC connection between two QPs. First on the receive side, I post two recv WRs, say recv_wr1 and recv_wr2. Then on the send side, I post two send WRs, say send_wr1, send_wr2. My question is, is there any possibility that send_wr2 finishes before send_wr1? What about the receive side? Is is possible that recv_wr2 is finished before recv_wr1?

Thanks,
Jiajun

It seems that the descriptions of IBV_WR_ATOMIC_CMP_AND_SWP and IBV_WR_ATOMIC_FETCH_AND_ADD are swapped.

Dotan,

You're writing regarding the inline data that "the low-level driver (i.e. CPU) will read the data and not the RDMA device". Is this correct for the both sides? I.e., on the responding side, will the HCA perform DMA for the inlined data, or will CPU handle it?

Thanks a lot for your assistance.

Hi Dotan,
Great website. Thanks for all the work.
Question about posting WRs. If I post a WR to a WQ, does a copy of the WR get made so that after the ibv_post_send() completes, I am free to overwrite that WR for my own purposes? Or is just a pointer to that WR posted to the WQ and I have to keep it intact until the completion occurs. It tried to find the internal representation of the WQs to see if I could deduce the answer myself, but no luck.

Hi Dotan,

This is a fantastic website for RDMA learners! I have a question regarding on the atomic operations. That is, how does the RDMA atomic operations (FetchAdd & CmpSwap) implemented? I guess there should be a locking mechanism that comes to work once the atomic operations are performed on some memory buffer. Is the lock implemented on the network (RNIC?), on the specific memory buffer, on the memory bus, or somewhere else?

Thanks in advance!

Henry

Hii Dotan,
I am trying to post send request in a queue that is already full. and I am getting some error (ENOMEM). So I put some sleep time and again post same request but it is again throwing same error. (Consider that after sleep time send queue is not full)

Hii Dotan,
First of all thanks for all your help, Finally my code is working and currently I am getting 3 times better performance for RDMA compare to UDP. I am having few more question that how much improvement(max) can we suppose with RDMA as compare to udp. Currently I am using only channel semantics, is there any good chances to improve if I use memory semantics also??

hi Dotan
(at the Target side) When I'm doing a RDMA-READ with 4 wr, each wr have 1 sge (4KB), the initiator will easyly crush or the /dev/sdxx dispear. (While doing RDMA-WRITE is fine.)
I've set the wr's rkey and increase remote_addr by 4096, any suggest?

Thanks
Zhang Yue
ps:
for(k = 1; k cache_req.sglist_size; k++)
{
multi_wr[k] = rdmad->send_wr; // copy struct

multi_wr[k].next = &multi_wr[k+1];
multi_wr[k].sg_list = &task->rdma_sge[k];
multi_wr[k].send_flags = 0; //zy: should be 0. otherwize will free task multi times
multi_wr[k].wr.rdma.remote_addr += (4096 * k);

task->rdma_sge[k].addr = tgt_phy2virt(task->cache_req.sglist[k].addr);
task->rdma_sge[k].length = task->cache_req.sglist[k].len;
task->rdma_sge[k].lkey = get_cache_buf_lkey(task->conn->dev, task->cache_req.sglist[k].addr);

}

// insert to list
multi_wr[k-1].next = rdmad->send_wr.next;
rdmad->send_wr.next = &multi_wr[1];

task->task_multi_wr = multi_wr;

//this sge.length mark the total length, will be use at iser_rdma_rd_comp_complete_handler
rdmad->sge.length = task->rdma_rd_sz;

// so we need to place the first wr's sge to other place
rdmad->send_wr.sg_list = task->rdma_sge;
task->rdma_sge[0].addr = tgt_phy2virt(task->cache_req.sglist[0].addr);
task->rdma_sge[0].length = task->cache_req.sglist[0].len;
task->rdma_sge[0].lkey = get_cache_buf_lkey(task->conn->dev, task->cache_req.sglist[0].addr);

Hi Dotan,

I want to post a request, but I want that the remote QP discards this request as soon as it receives it. This is because I want to send a dummy packet when I am in the REARM state in the QPs in order to reach the ARMED state (this is because it is needed an incoming packet for this transition).

I am using the below configuration and it seems to be working, but I would like to know if you think that this could be a generic approach for any situation or not:

struct ibv_send_wr wr;
struct ibv_send_wr *bad_wr;

memset(&wr, 0, sizeof(wr));
wr.wr_id = 0;
wr.sg_list = NULL;
wr.num_sge = 0;
wr.opcode = 0;
wr.send_flags = 0;

if (ibv_post_send(ctx->id[num_qp]->qp, &wr, &bad_wr)) {
fprintf(stderr, "Error, ibv_post_send() failed\n");
return -1;
}

Best regards,
Jesus Camacho

Hello Dotan,

I have a quick question. What happens if the local node calls ibv_post_send() with opcode ibv_wr_send before the remote node calls ibv_post_recv()?

Thanks!

Hi Dotan:

Nice to meet you. I'm from China. My English is not very good. Recently I have learn somthing about RDMA. But I met a problem:

This is my test program:
server code :

/*
* Copyright (C) fuchencong@163.com
*/

#include
#include
#include
#include
#include
#include

#define VERB_ERR(verb, ret) \
fprintf(stderr, "%s returned %d errno %d\n", verb, ret, errno)

#define MB 1024 * 1024

/* Default parameter values */
#define DEFAULT_PORT "51216"
#define DEFAULT_MSG_COUNT 100
#define DEFAULT_MSG_LENGTH MB

/* Resources used in the example */
struct context
{
char *server_name;
char *server_port;
unsigned int msg_count;
unsigned int msg_length;
/* Resources */
struct rdma_cm_id *id;
struct rdma_cm_id *listen_id;
struct ibv_mr *recv_mr;
char *recv_buf;
};

int
reg_mem(struct context *ctx)
{
ctx->recv_buf = (char *) malloc(ctx->msg_length);
memset(ctx->recv_buf, 0x00, ctx->msg_length);

ctx->recv_mr = rdma_reg_msgs(ctx->id, ctx->recv_buf, ctx->msg_length);
if (!ctx->recv_mr) {
VERB_ERR("rdma_reg_msgs", -1);
return -1;
}

return 0;
}

int
getaddrinfo_and_create_ep(struct context *ctx)
{
int ret;
struct rdma_addrinfo *rai, hints;
struct ibv_qp_init_attr qp_init_attr;

memset(&hints, 0, sizeof (hints));
hints.ai_port_space = RDMA_PS_TCP;
hints.ai_flags = RAI_PASSIVE; /* this makes it a server */

printf("rdma_getaddrinfo\n");
ret = rdma_getaddrinfo(ctx->server_name, ctx->server_port, &hints, &rai);
if (ret) {
VERB_ERR("rdma_getaddrinfo", ret);
return ret;
}

memset(&qp_init_attr, 0, sizeof (qp_init_attr));
qp_init_attr.cap.max_send_wr = 1;
qp_init_attr.cap.max_recv_wr = 1;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;

printf("rdma_create_ep\n");
ret = rdma_create_ep(&ctx->id, rai, NULL, &qp_init_attr);
if (ret) {
VERB_ERR("rdma_create_ep", ret);
return ret;
}

rdma_freeaddrinfo(rai);

return 0;
}

int
get_connect_request(struct context *ctx)
{
int ret;
printf("rdma_listen\n");

ret = rdma_listen(ctx->id, 4);
if (ret) {
VERB_ERR("rdma_listen", ret);
return ret;
}

ctx->listen_id = ctx->id;
printf("rdma_get_request\n");
ret = rdma_get_request(ctx->listen_id, &ctx->id);
if (ret) {
VERB_ERR("rdma_get_request", ret);
return ret;
}

if (ctx->id->event->event != RDMA_CM_EVENT_CONNECT_REQUEST) {
printf("unexpected event: %s", \
rdma_event_str(ctx->id->event->event));
return ret;
}

return 0;
}

int
establish_connection(struct context *ctx)
{
int ret;
struct rdma_conn_param conn_param;

/* post a receive to catch the first send */
ret = rdma_post_recv(ctx->id, NULL, ctx->recv_buf, ctx->msg_length,
ctx->recv_mr);
if (ret) {
VERB_ERR("rdma_post_recv", ret);
return ret;
}

memset(&conn_param, 0, sizeof (conn_param));
conn_param.responder_resources = 2;
conn_param.initiator_depth = 2;
conn_param.retry_count = 5;
conn_param.rnr_retry_count = 5;

printf("rdma_accept\n");
ret = rdma_accept(ctx->id, &conn_param);
if (ret) {
VERB_ERR("rdma_accept", ret);
return ret;
}

return 0;
}

int
recv_msg(struct context *ctx)
{
int ret;
struct ibv_wc wc;

ret = rdma_get_recv_comp(ctx->id, &wc);
if (ret id, NULL, ctx->recv_buf, ctx->msg_length,
ctx->recv_mr);
if (ret) {
VERB_ERR("rdma_post_recv", ret);
return ret;
}

return 0;
}

int
main(int argc, char** argv)
{
int ret, op, i, recv_cnt;
struct context ctx;
struct ibv_qp_attr qp_attr;

memset(&ctx, 0, sizeof (ctx));
memset(&qp_attr, 0, sizeof (qp_attr));

ctx.server_port = DEFAULT_PORT;
ctx.msg_count = DEFAULT_MSG_COUNT;
ctx.msg_length = DEFAULT_MSG_LENGTH;

while ((op = getopt(argc, argv, "a:p:c:l:")) != -1) {
switch (op) {
case 'a':
ctx.server_name = optarg;
break;
case 'p':
ctx.server_port = optarg;
break;
case 'c':
ctx.msg_count = atoi(optarg);
break;
case 'l':
ctx.msg_length = atoi(optarg) * MB;
break;
default:
printf("usage: %s [-s or -a required]\n", argv[0]);
printf("\t[-a ip_address]\n");
printf("\t[-p port_number]\n");
printf("\t[-c msg_count]\n");
printf("\t[-l msg_length]\n");
exit(1);
}
}

printf("address: %s\n", (!ctx.server_name) ? "NULL" : ctx.server_name);
printf("port: %s\n", ctx.server_port);
printf("count: %d\n", ctx.msg_count);
printf("length: %d bytes\n", ctx.msg_length);
printf("\n");

ret = getaddrinfo_and_create_ep(&ctx);
if (ret) {
goto out;
}

ret = get_connect_request(&ctx);
if (ret) {
goto out;
}

ret = reg_mem(&ctx);
if (ret) {
goto out;
}

ret = establish_connection(&ctx);

recv_cnt = 0;
for (i = 0; i < ctx.msg_count; i++) {
if (recv_msg(&ctx)) {
break;
}
++recv_cnt;
}
printf("recv %d messages, each message is %d bytes\n", \
recv_cnt, ctx.msg_length);

rdma_disconnect(ctx.id);

out:
if (ctx.recv_mr) {
rdma_dereg_mr(ctx.recv_mr);
}

if (ctx.id) {
rdma_destroy_ep(ctx.id);
}

if (ctx.listen_id) {
rdma_destroy_ep(ctx.listen_id);
}

if (ctx.recv_buf) {
free(ctx.recv_buf);
}

return ret;
}

client code:

/*
* Copyright (C) fuchencong@163.com
*/

#include
#include
#include
#include
#include
#include

#define VERB_ERR(verb, ret) \
fprintf(stderr, "%s returned %d errno %d\n", verb, ret, errno)

#define MB 1024 * 1024

/* Default parameter values */
#define DEFAULT_PORT "51216"
#define DEFAULT_MSG_COUNT 100
#define DEFAULT_MSG_LENGTH MB
#define DEFAULT_MSEC_DELAY 500

/* Resources used in the example */
struct context
{
char *server_name;
char *server_port;
unsigned int msg_count;
unsigned int msg_length;
/* Resources */
struct rdma_cm_id *id;
struct ibv_mr *send_mr;
char *send_buf;
};

int
reg_mem(struct context *ctx)
{
ctx->send_buf = (char *) malloc(ctx->msg_length);
memset(ctx->send_buf, 'a', ctx->msg_length);

ctx->send_mr = rdma_reg_msgs(ctx->id, ctx->send_buf, ctx->msg_length);
if (!ctx->send_mr) {
VERB_ERR("rdma_reg_msgs", -1);
return -1;
}

return 0;
}

int
getaddrinfo_and_create_ep(struct context *ctx)
{
int ret;
struct rdma_addrinfo *rai, hints;
struct ibv_qp_init_attr qp_init_attr;

memset(&hints, 0, sizeof (hints));
hints.ai_port_space = RDMA_PS_TCP;

printf("rdma_getaddrinfo\n");
ret = rdma_getaddrinfo(ctx->server_name, ctx->server_port, &hints, &rai);
if (ret) {
VERB_ERR("rdma_getaddrinfo", ret);
return ret;
}

memset(&qp_init_attr, 0, sizeof (qp_init_attr));
qp_init_attr.cap.max_send_wr = 1;
qp_init_attr.cap.max_recv_wr = 1;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;

printf("rdma_create_ep\n");
ret = rdma_create_ep(&ctx->id, rai, NULL, &qp_init_attr);
if (ret) {
VERB_ERR("rdma_create_ep", ret);
return ret;
}

rdma_freeaddrinfo(rai);

return 0;
}

int
establish_connection(struct context *ctx)
{
int ret;
struct rdma_conn_param conn_param;

memset(&conn_param, 0, sizeof (conn_param));
conn_param.private_data_len = sizeof (int);
conn_param.responder_resources = 2;
conn_param.initiator_depth = 2;
conn_param.retry_count = 5;
conn_param.rnr_retry_count = 5;

printf("rdma_connect\n");
ret = rdma_connect(ctx->id, &conn_param);
if (ret) {
VERB_ERR("rdma_connect", ret);
return ret;
}

if (ctx->id->event->event != RDMA_CM_EVENT_ESTABLISHED) {
printf("unexpected event: %s",
rdma_event_str(ctx->id->event->event));
return -1;
}

return 0;
}

int
send_msg(struct context *ctx)
{
int ret;
struct ibv_wc wc;

ret = rdma_post_send(ctx->id, NULL, ctx->send_buf, ctx->msg_length,
ctx->send_mr, IBV_SEND_SIGNALED);
if (ret) {
VERB_ERR("rdma_send_recv", ret);
return ret;
}

ret = rdma_get_send_comp(ctx->id, &wc);
if (ret < 0) {
VERB_ERR("rdma_get_send_comp", ret);
return ret;
}

return 0;
}

int
main(int argc, char** argv)
{
int ret, op, i, send_cnt;
struct context ctx;
struct ibv_qp_attr qp_attr;

memset(&ctx, 0, sizeof (ctx));
memset(&qp_attr, 0, sizeof (qp_attr));

ctx.server_port = DEFAULT_PORT;
ctx.msg_count = DEFAULT_MSG_COUNT;
ctx.msg_length = DEFAULT_MSG_LENGTH;

while ((op = getopt(argc, argv, "a:p:c:l:")) != -1) {
switch (op) {
case 'a':
ctx.server_name = optarg;
break;
case 'p':
ctx.server_port = optarg;
break;
case 'c':
ctx.msg_count = atoi(optarg);
break;
case 'l':
ctx.msg_length = atoi(optarg) * MB;
break;
default:
printf("usage: %s [-s or -a required]\n", argv[0]);
printf("\t[-a ip_address]\n");
printf("\t[-p port_number]\n");
printf("\t[-c msg_count]\n");
printf("\t[-l msg_length]\n");
exit(1);
}
}

printf("address: %s\n", (!ctx.server_name) ? "NULL" : ctx.server_name);
printf("port: %s\n", ctx.server_port);
printf("count: %d\n", ctx.msg_count);
printf("length: %d bytes\n", ctx.msg_length);
printf("\n");

if (!ctx.server_name) {
printf("server address must be specified for client\n");
exit(1);
}

ret = getaddrinfo_and_create_ep(&ctx);
if (ret) {
goto out;
}

ret = reg_mem(&ctx);
if (ret) {
goto out;
}

ret = establish_connection(&ctx);

send_cnt = 0;
for (i = 0; i < ctx.msg_count; i++) {
if (send_msg(&ctx)) {
break;
}
++send_cnt;
}
printf("send %d messages, each message is %d bytes\n", \
send_cnt, ctx.msg_length);

rdma_disconnect(ctx.id);

out:
if (ctx.send_mr) {
rdma_dereg_mr(ctx.send_mr);
}

if (ctx.id) {
rdma_destroy_ep(ctx.id);
}

if (ctx.send_buf) {
free(ctx.send_buf);
}

return ret;
}

What I can't understand is that sometimes this program takes 1 minite to send 1G data and sometimes it only needs 0.2 seconds。 So it's not very stable.

I really don't know why. Can you give me some advice?
Thank you!

Thanks Dotan, I figured it out. Something wrong in another module...

Hi Dotan.

Thank you for such a guideline of rdma programing!

And, I have some trouble about IBV_WR_SEND in UD. I use doorbell batching to post my sends (just like wr[i].next = &wr[i+1]). However, only the data of the lattest wr in the batching is received. I am sure that there is no error thrown in my code because if I replace IBV_WR_SEND with IBV_WR_SEND_WITH_IMMEDIATE it works for the same code, the headers arrive correctly. Also, if I just use a post_send for each wr, it works. I think something in the sender side is wrong.

Hope that you can give me some advice!

Thanks!

Hi,Dotan,I have read many of your articles to learn RDMA programming.
Now I get some problems and try to search result from RDMA_Aware_Programming_User_Manual.pdf (Version 1.7) and the IB Specification Vol 1-Release-1.3-2015-03-03.pdf,but haven't found the result.So I have to turn to you for help.The problem is When I post work request to queuepair,the NIC got notification and fetch the work request from memory to NIC cache by DMA,but when NIC send the data contained in the work reqeust to cabel,does it need to fetch the queuepair information to NIC cache?I know that NIC cache stores the queuepair data,memory address translation data and some network data,but when NIC send data,is the queuepair information necessary?