Chapter 4. Configuring the core RDMA subsystem
The rdma service configuration manages the network protocols and communication standards such as InfiniBand, iWARP, and RoCE.
4.1. Renaming IPoIB devices using systemd link file
By default, the kernel names Internet Protocol over InfiniBand (IPoIB) devices, for example, ib0, ib1, and so on. To avoid conflicts, create a systemd link file to create persistent and meaningful names such as mlx4_ib0.
Prerequisites
- You have installed an InfiniBand device.
Procedure
Display the hardware address of the device
ib0:# ip addr show ib0 7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 256 link/infiniband 80:00:0a:28:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7b:e1:b1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff altname ibp7s0 altname ibs2 inet 172.31.0.181/24 brd 172.31.0.255 scope global dynamic noprefixroute ib0 valid_lft 2899sec preferred_lft 2899sec inet6 fe80::f652:1403:7b:e1b1/64 scope link noprefixroute valid_lft forever preferred_lft foreverFor naming the interface with MAC address
80:00:0a:28:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7b:e1:b1tomlx4_ib0, create the/etc/systemd/network/70-custom-ifnames.linkfile with following contents:[Match] MACAddress=80:00:0a:28:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7b:e1:b1 [Link] Name=mlx4_ib0
This link file matches a MAC address and renames the network interface to the name set in the
Nameparameter.
Verification
Reboot the host:
# rebootVerify that the device with the MAC address you specified in the link file has been assigned to
mlx4_ib0:# ip addr show mlx4_ib0 7: mlx4_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UP group default qlen 256 link/infiniband 80:00:0a:28:fe:80:00:00:00:00:00:00:f4:52:14:03:00:7b:e1:b1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff altname ibp7s0 altname ibs2 inet 172.31.0.181/24 brd 172.31.0.255 scope global dynamic noprefixroute mlx4_ib0 valid_lft 2899sec preferred_lft 2899sec inet6 fe80::f652:1403:7b:e1b1/64 scope link noprefixroute valid_lft forever preferred_lft forever
Additional resources
-
systemd.link(5)man page
4.2. Increasing the amount of memory that users are allowed to pin in the system
Remote direct memory access (RDMA) operations require the pinning of physical memory. As a consequence, the kernel is not allowed to write memory into the swap space. If a user pins too much memory, the system can run out of memory, and the kernel terminates processes to free up more memory. Therefore, memory pinning is a privileged operation.
If non-root users need to run large RDMA applications, it is necessary to increase the amount of memory to maintain pages in primary memory pinned all the time.
Procedure
As the
rootuser, create the file/etc/security/limits.confwith the following contents:@rdma soft memlock unlimited @rdma hard memlock unlimited
Verification
Log in as a member of the
rdmagroup after editing the/etc/security/limits.conffile.Note that Red Hat Enterprise Linux applies updated
ulimitsettings when the user logs in.Use the
ulimit -lcommand to display the limit:$ ulimit -l unlimitedIf the command returns
unlimited, the user can pin an unlimited amount of memory.
Additional resources
-
limits.conf(5)man page
4.3. Enabling NFS over RDMA (NFSoRDMA)
In Red Hat Enterprise Linux 9, Remote direct memory access (RDMA) service on RDMA-capable hardware provides Network File System (NFS) protocol support for high-speed file transfer over the network.
Procedure
Install the
rdma-corepackage:# dnf install rdma-coreVerify the lines with
xprtrdmaandsvcrdmaare not commented out in the/etc/rdma/modules/rdma.conffile:# NFS over RDMA client support xprtrdma # NFS over RDMA server support svcrdma
On the NFS server, create directory
/mnt/nfsordmaand export it to/etc/exports:# mkdir /mnt/nfsordma # echo "/mnt/nfsordma *(fsid=0,rw,async,insecure,no_root_squash)" >> /etc/exports
On the NFS client, mount the nfs-share with server IP address, for example,
172.31.0.186:# mount -o rdma,port=20049 172.31.0.186:/mnt/nfs-share /mnt/nfsRestart the
nfs-serverservice:# systemctl restart nfs-server
Additional resources