9.3. Configuring Soft-RoCE

RoCE can be implemented both in the hardware and in the software. Soft-RoCE is the software implementation of the RDMA transport.

Prerequisites

Since Red Hat Enterprise Linux 7.4, the Soft-RoCE driver is already merged into the kernel. The user-space driver also is merged into the rdma-core package. Soft-RoCE is also known as RXE. To start, stop and configure RXE, use the rxe_cfg script. To view options for rxe_cfg, enter rxe_cfg help.

Procedure 9.2. Configuring Soft-RoCE

  1. As the root user, enter the following command to display the current configuration status of RXE:
    ~]# rxe_cfg
      rdma_rxe module not loaded
      Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
      igb_1       yes   igb
      mlx4_1      no    mlx4_en
      mlx4_2      no    mlx4_en
  2. To load the RXE kernel module and start RXE, enter as root:
    ~]# rxe_cfg start
      Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
      igb_1       yes   igb
      mlx4_1      no    mlx4_en
      mlx4_2      no    mlx4_en
    Optionally, to verify that the RXE kernel module is loaded, enter:
    ~]# lsmod |grep rdma_rxe
      rdma_rxe              111129  0
      ip6_udp_tunnel         12755  1 rdma_rxe
      udp_tunnel             14423  1 rdma_rxe
      ib_core               236827  15 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_rxe,rdma_ucm,ib_ipoib,ib_isert
    
  3. Before adding a new RXE device over an Ethernet interface, the corresponding interface should be opened and has a valid IP address assigned. To add a new RXE device, for example igb_1:
    ~]# rxe_cfg add igb_1
    ~]# rxe_cfg status
      Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
      igb_1       yes   igb                               rxe0  1024  (3)
      mlx4_1      no    mlx4_en
      mlx4_2      no    mlx4_en
    The rxe0 in the RDEV column indicates that rxe is enabled for the igb_1 device.
  4. To verify the status of an RXE device, use the ibv_devices command:
    ~]# ibv_devices
      device                 node GUID
      ------              ----------------
      mlx4_0              0002c90300b3cff0
      rxe0                a2369ffffe018294
    Alternatively, enter the ibstat for a detailed status:
    ~]# ibstat rxe0
    CA 'rxe0'
        CA type:
        Number of ports: 1
        Firmware version:
        Hardware version:
        Node GUID: 0xa2369ffffe018294
        System image GUID: 0x0000000000000000
        Port 1:
            State: Active
            Physical state: LinkUp
            Rate: 2.5
            Base lid: 0
            LMC: 0
            SM lid: 0
            Capability mask: 0x00890000
            Port GUID: 0xa2369ffffe018294
            Link layer: Ethernet

Removing an RXE device

If you want to remove an RXE device, enter:
~]# rxe_cfg remove igb_1

Verifying Connectivity of an RXE device

The following examples show how to verify connectivity of an RXE device on the server and client side.

Example 9.1. Verifying Connectivity of an RXE device on the Server Side

~]$ ibv_rc_pingpong -d rxe0 -g 0
  local address:  LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
  remote address: LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
8192000 bytes in 0.05 seconds = 1244.06 Mbit/sec
1000 iters in 0.05 seconds = 52.68 usec/iter

Example 9.2. Verifying Connectivity of an RXE device on the Client Side

~]$ ibv_rc_pingpong -d rxe0 -g 0 172.31.40.4
  local address:  LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
  remote address: LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
 8192000 bytes in 0.05 seconds = 1245.72 Mbit/sec
 1000 iters in 0.05 seconds = 52.61 usec/iter