Menu Close

Chapter 4. Configuring the core RDMA subsystem

This section describes how to configure the rdma service and increase the amount of memory that users are allowed to pin in the system.

4.1. Renaming IPoIB devices

By default, the kernel names Internet Protocol over InfiniBand (IPoIB) devices, for example, ib0, ib1, and so on. To avoid conflicts, Red Hat recommends creating a rule in the udev device manager to create persistent and meaningful names such as mlx4_ib0.

Prerequisites

  • An InfiniBand device is installed

Procedure

  1. Display the hardware address of the device ib0:

    # ip link show ib0
    8: ib0: >BROADCAST,MULTICAST,UP,LOWER_UP< mtu 65520 qdisc pfifo_fast state UP mode DEFAULT qlen 256
        link/infiniband 80:00:02:00:fe:80:00:00:00:00:00:00:00:02:c9:03:00:31:78:f2 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

    The last eight bytes of the address are required to create a udev rule in the next step.

  2. To configure a rule that renames the device with the 00:02:c9:03:00:31:78:f2 hardware address to mlx4_ib0, edit the /etc/udev/rules.d/70-persistent-ipoib.rules file and add an ACTION rule:

    ACTION=="add", SUBSYSTEM=="net", DRIVERS=="?*", ATTR{type}=="32", ATTR{address}=="?*00:02:c9:03:00:31:78:f2", NAME="mlx4_ib0"
  3. Reboot the host:

    # reboot

Additional resources

4.2. Increasing the amount of memory that users are allowed to pin in the system

Remote direct memory access (RDMA) operations require the pinning of physical memory. As a consequence, the kernel is not allowed to write memory into the swap space. If a user pins too much memory, the system can run out of memory, and the kernel terminates processes to free up more memory. Hence, memory pinning is a privileged operation.

If non-root users run large RDMA applications, it is necessary to increase the amount of memory these users can pin in the system. This section describes how to configure an unlimited amount of memory for the rdma group.

Procedure

  • As the root user, create the file /etc/security/limits.conf with following contents:

    @rdma soft memlock unlimited
    @rdma hard memlock unlimited

Verification

  1. Log in as a member of the rdma group after editing the /etc/security/limits.conf file.

    Note that Red Hat Enterprise Linux applies updated ulimit settings when the user logs in.

  2. Use the ulimit -l command to display the limit:

    $ ulimit -l
    unlimited

    If the command returns unlimited, the user can pin an unlimited amount of memory.

Additional resources

  • limits.conf(5) man page

4.3. Configuring the rdma service

The rdma service manages the stack in the kernel. If Red Hat Enterprise Linux detects InfiniBand, iWARP, or RoCE devices and configuration file of the same reside at the /etc/rdma/modules/*, the udev device manager instructs systemd to start the rdma service. By default, /etc/rdma/modules/rdma.conf configures and loads these services.

Procedure

  1. Edit the /etc/rdma/modules/rdma.conf file and set the variable to yes that you want to enable:

    # Load IPoIB
    IPOIB_LOAD=yes
    # Load SRP (SCSI Remote Protocol initiator support) module
    SRP_LOAD=yes
    # Load SRPT (SCSI Remote Protocol target support) module
    SRPT_LOAD=yes
    # Load iSER (iSCSI over RDMA initiator support) module
    ISER_LOAD=yes
    # Load iSERT (iSCSI over RDMA target support) module
    ISERT_LOAD=yes
    # Load RDS (Reliable Datagram Service) network protocol
    RDS_LOAD=no
    # Load NFSoRDMA client transport module
    XPRTRDMA_LOAD=yes
    # Load NFSoRDMA server transport module
    SVCRDMA_LOAD=no
    # Load Tech Preview device driver modules
    TECH_PREVIEW_LOAD=no
  2. Restart the rdma service:

    # systemctl restart rdma

4.4. Enabling NFS over RDMA (NFSoRDMA)

Remote direct memory access (RDMA) service works automatically on RDMA-capable hardware in Red Hat Enterprise Linux 8.

Procedure

  1. Install the rdma-core package:

    # yum install rdma-core
  2. Verify the lines with xprtrdma and svcrdma are commented out in the /etc/rdma/modules/rdma.conf file:

    # NFS over RDMA client support
    xprtrdma
    # NFS over RDMA server support
    svcrdma
  3. On the NFS server, create directory /mnt/nfsordma and export it to /etc/exports:

    # mkdir /mnt/nfsordma
    # echo "/mnt/nfsordma *(fsid=0,rw,async,insecure,no_root_squash)" >> /etc/exports
  4. On the NFS client, mount the nfs-share with server IP address, for example, 172.31.0.186:

    # mount -o rdma,port=20049 172.31.0.186:/mnt/nfs-share /mnt/nfs
  5. Restart the nfs-server service:

    # systemctl restart nfs-server

Additional resources