Chapter 4. Configuring the core RDMA subsystem

The rdma service configuration manages the network protocols and communication standards such as InfiniBand, iWARP, and RoCE.

4.2. Increasing the amount of memory that users are allowed to pin in the system

Remote direct memory access (RDMA) operations require the pinning of physical memory. As a consequence, the kernel is not allowed to write memory into the swap space. If a user pins too much memory, the system can run out of memory, and the kernel terminates processes to free up more memory. Therefore, memory pinning is a privileged operation.

If non-root users need to run large RDMA applications, it is necessary to increase the amount of memory to maintain pages in primary memory pinned all the time.


  • As the root user, create the file /etc/security/limits.conf with the following contents:

    @rdma soft memlock unlimited
    @rdma hard memlock unlimited


  1. Log in as a member of the rdma group after editing the /etc/security/limits.conf file.

    Note that Red Hat Enterprise Linux applies updated ulimit settings when the user logs in.

  2. Use the ulimit -l command to display the limit:

    $ ulimit -l

    If the command returns unlimited, the user can pin an unlimited amount of memory.

Additional resources

  • limits.conf(5) man page

4.3. Enabling NFS over RDMA (NFSoRDMA)

In Red Hat Enterprise Linux 9, Remote direct memory access (RDMA) service on RDMA-capable hardware provides Network File System (NFS) protocol support for high-speed file transfer over the network.


  1. Install the rdma-core package:

    # dnf install rdma-core
  2. Verify the lines with xprtrdma and svcrdma are not commented out in the /etc/rdma/modules/rdma.conf file:

    # NFS over RDMA client support
    # NFS over RDMA server support
  3. On the NFS server, create directory /mnt/nfsordma and export it to /etc/exports:

    # mkdir /mnt/nfsordma
    # echo "/mnt/nfsordma *(fsid=0,rw,async,insecure,no_root_squash)" >> /etc/exports
  4. On the NFS client, mount the nfs-share with server IP address, for example,

    # mount -o rdma,port=20049 /mnt/nfs
  5. Restart the nfs-server service:

    # systemctl restart nfs-server

Additional resources