Allow Infiniband and RDMA Operations To Run Without Root Priveledges

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8.x
  • InfiniBand

Issue

  • I would like to allow direct communication between InfiniBand enabled systems without a switch.
  • I would like to use the command ibping without root permissions in order to verify communication between two systems.

Resolution

  • To enable the use of ibping, the non-root user must be added to the rdma group.

  1. On the client as root, we must group add rdma and ensure that the non-root user is group rdma.

    [root@ibclient net]# cd /dev/infiniband

[root@ibclient net]# chown root:rdma umad*

[root@ibclient net]# chmod 775 umad*

  2. On the client login to the non-root user.

    [root@ibclient infiniband]# su - rh

[rh@ibclient ~]$ ibping -c 10000 -f -C mlx5_1 -P 1 -L 2
--- ibserver.cee.lab.eng.bos.redhat.com.(none) (Lid 2) ibping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 2523 ms
rtt min/avg/max = 0.039/0.252/0.473 ms

  • For these changes to take effect, users that have been added to the rdma group must completely log out and back in again.

  • The command ibping now works with the non-root user that is included in the rdma group.

Diagnostic Steps

  • As tested in the Red Hat labs

  1. Setting up the reproducer. On the server, as root.

    [root@ibserver ~]# ibping -S -C mlx5_1 -P 1

  2. On the client, as root.

    [root@ibclient net]# ibping -c 10000 -f -C mlx5_1 -P 1 -L 2

--- ibserver.cee.lab.eng.bos.redhat.com.(none) (Lid 2) ibping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 2280 ms
rtt min/avg/max = 0.038/0.227/0.490 ms

  3. Reproducing the problem. On the server, as the non-root user.

    [rh@ibserver ~]$ ibping -S -C mlx5_1 -P 1
ibwarn: [31114] mad_rpc_open_port: can't open UMAD port (mlx5_1:1)
ibping: iberror: failed: Failed to open 'mlx5_1' port '1'

  4. On the client, as the non-root user.

    [rh@ibclient ~]$ ibping -c 10000 -f -C mlx5_1 -P 1 -L 2
ibwarn: [23953] mad_rpc_open_port: can't open UMAD port (mlx5_1:1)
ibping: iberror: failed: Failed to open 'mlx5_1' port '1'

total 0
crw------- 1 root rdma 231,   0 Nov 17 13:57 umad0
crw------- 1 root rdma 231,   1 Nov 17 13:57 umad1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments