int ibv_fork_init(void);

Prerequisite

ibv_fork_init() should be called before calling any other function in libibverbs.

Description

ibv_fork_init() initializes libibverbs's data structures to handle the fork() function calls correctly and avoid data corruption, whether fork() is called explicitly or implicitly (such as in system(), popen(), etc.).

In RDMA technologies, the RDMA device is aware to the virtual->physical address mapping. When a process calls the fork() function explicitly or implicitly, due to the Copy-On-Write policy of the Linux kernel, both parent and child processes virtual pages will be mapped to the same physical memory pages. When one of them (the first) will write to such a page, it will get a new physical page, with the same content. The RDMA device, which isn't aware that the virtual->physical translation was changed to that process, may try to access the old physical address (which now may even be used by other process or the kernel). Which may lead to a data corruption or even worse - a kernel panic.

To prevent this problem from happening, libibverbs added fork() protection support; it instructs the Linux kernel to keep the original physical pages mapped to the parent process, regardless of which of the two processes writes to the page first.

The fork() support in libibverbs isn't full and it assumes that only the parent process will do the RDMA operations. If one of the child processes will try to perform RDMA operations, it may experience various problems.

It is not necessary to use this function if all parent process threads are always blocked until all child processes end or change address spaces via an exec() operation.

If fork() will be used without calling ibv_fork_init(), one may experience data corruption, segmentation fault, missing Work Request completions or any other phenomena.

ibv_fork_init() works on Linux kernels supporting the MADV_DONTFORK
flag for madvise() (2.6.17 and higher).

Calling ibv_fork_init() will reduce performance due to an extra system call for every memory registration, and the extra memory allocated to track memory regions. The precise performance impact depends on the workload and usually will not be significant.

Setting one of the environment variables RDMAV_FORK_SAFE or IBV_FORK_SAFE to any value (libibverbs only check if one of the environment variables was defined and completely ignores the value)  has the same effect as calling ibv_fork_init().

Setting the environment variable RDMAV_HUGEPAGES_SAFE tells the library to check the underlying page size used by the kernel for memory regions. This is required if an application uses huge pages either directly or indirectly via a library such as libhugetlbfs.

There isn't any function call that has the same effect as setting the environment variable RDMAV_HUGEPAGES_SAFE.

Setting RDMAV_HUGEPAGES_SAFE adds further overhead to all memory registrations.

Parameters

None

Return Values

Examples

Setting environment variable for fork() protection (in bash), all the following lines have the same effect:

# export RDMAV_FORK_SAFE
# export RDMAV_FORK_SAFE=0
# export RDMAV_FORK_SAFE=1
# export RDMAV_FORK_SAFE="no"
# export RDMAV_FORK_SAFE="yes"
# export IBV_FORK_SAFE=0
# export IBV_FORK_SAFE=1
# export IBV_FORK_SAFE="no"

Setting environment variable for fork() protection when huge pages are used (in bash):

# export RDMAV_FORK_SAFE
# export RDMAV_HUGEPAGES_SAFE

Call function for fork() protection (in c source code):

int rc;
 
rc = ibv_fork_init()
if (rc)
        exit(rc);

FAQs

What will happen if I won't enable fork() support and I will still use fork()?

Bad things may happen, such as (but not limited to):

- Data corruption

- Missing Work Completions

- Segmentation fault

Does is matter if I call ibv_fork_init() or set the environment variable RDMAV_FORK_SAFE?

No, they care completely equivalent.

Does is matter which value I assign to the environment variable RDMAV_FORK_SAFE?

No, any value should do.

I'm using huge pages and I only enabled RDMAV_FORK_SAFE, will it fork for me?

No, any value should set the environment variable RDMAV_HUGEPAGES_SAFE too.

I don't know if I need fork() support, can I enable fork() support, just to be on the safe side?

Yes, you can. Just be aware that extra memory will be consumed when enabling fork support

I don't call explicitly to fork(), I only call to other system calls that may call fork(), do I still need to enable fork() support?

Yes, you need to enable fork() support, whether fork() is called directly or if any other system calls call fork() implicitly.

I only provide a library to my customer, and he calls fork(), does fork() support is still required?

Yes, fork() support needs to be enabled if the process calls fork(), it doesn't matter who calls it. If the user will sometimes use fork(), he can set the environment variable RDMAV_FORK_SAFE.

I called ibv_fork_init() and it failed, what does it mean?

It means that your kernel doesn't support madvise(), you should consult your support or update your kernel.

Can I inherit the RDMA resources after I call fork()?

The libibverbs has several file descriptor and after calling fork(), they still be available in the child process. However, it is highly recommended not to try to use them since it may cause bad things.

Written by: Dotan Barak on May 24, 2012.on January 11, 2019.