Chapter 4. Configuring the core RDMA subsystem

The rdma service configuration manages the network protocols and communication standards such as InfiniBand, iWARP, and RoCE.

4.2. Increasing the amount of memory that users are allowed to pin in the system

Remote direct memory access (RDMA) operations require the pinning of physical memory. As a consequence, the kernel is not allowed to write memory into the swap space. If a user pins too much memory, the system can run out of memory, and the kernel terminates processes to free up more memory. Therefore, memory pinning is a privileged operation.

If non-root users need to run large RDMA applications, it is necessary to increase the amount of memory to maintain pages in primary memory pinned all the time.

Procedure

  • As the root user, create the file /etc/security/limits.conf with the following contents: 

    @rdma soft memlock unlimited
@rdma hard memlock unlimited

Verification

  1. Log in as a member of the rdma group after editing the /etc/security/limits.conf file.

    Note that Red Hat Enterprise Linux applies updated ulimit settings when the user logs in.

  2. Use the ulimit -l command to display the limit: 

    $ ulimit -l
unlimited

    If the command returns unlimited, the user can pin an unlimited amount of memory.

Additional resources

  • limits.conf(5) man page

4.3. Enabling NFS over RDMA on an NFS server

Remote Direct Memory Access (RDMA) is a protocol that enables a client system to directly transfer data from the memory of a storage server into its own memory. This enhances storage throughput, decreases latency in data transfer between the server and client, and reduces CPU load on both ends. If both the NFS server and clients are connected over RDMA, clients can use NFSoRDMA to mount an exported directory.

Prerequisites

  • The NFS service is running and configured
  • An InfiniBand or RDMA over Converged Ethernet (RoCE) device is installed on the server.
  • IP over InfiniBand (IPoIB) is configured on the server, and the InfiniBand device has an IP address assigned.

Procedure

  1. Install the rdma-core package: 

    # dnf install rdma-core

  2. If the package was already installed, verify that the xprtrdma and svcrdma modules in the /etc/rdma/modules/rdma.conf file are uncommented: 

    # NFS over RDMA client support
xprtrdma
# NFS over RDMA server support
svcrdma

  3. Optional. By default, NFS over RDMA uses port 20049. If you want to use a different port, set the rdma-port setting in the [nfsd] section of the /etc/nfs.conf file: 

    rdma-port=_<port>_

  4. Open the NFSoRDMA port in firewalld: 

    # firewall-cmd --permanent --add-port={20049/tcp,20049/udp}
# firewall-cmd --reload

    Adjust the port numbers if you set a different port than 20049.

  5. Restart the nfs-server service: 

    # systemctl restart nfs-server

Verification

  1. On a client with InfiniBand hardware, perform the following steps:

    1. Install the following packages: 

      # dnf install nfs-utils rdma-core

    2. Mount an exported NFS share over RDMA: 

      # mount -o rdma server.example.com:/nfs/projects/ /mnt/

      If you set a port number other than the default (20049), pass port=<port_number> to the command: 

      # mount -o rdma,port=<port_number> server.example.com:/nfs/projects/ /mnt/

    3. Verify that the share was mounted with the rdma option: 

      # mount | grep "/mnt"
server.example.com:/nfs/projects/ on /mnt type nfs (...,proto=rdma,...)

Additional resources