13.3. Configuring Soft-RoCE

Procedure 13.2. Configuring Soft-RoCE

1. As the root user, enter the following command to display the current configuration status of RXE:

   ~]# rxe_cfg
     rdma_rxe module not loaded
     Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
     igb_1       yes   igb
     mlx4_1      no    mlx4_en
     mlx4_2      no    mlx4_en

2. To load the RXE kernel module and start RXE, enter as root:

   ~]# rxe_cfg start
     Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
     igb_1       yes   igb
     mlx4_1      no    mlx4_en
     mlx4_2      no    mlx4_en

   Optionally, to verify that the RXE kernel module is loaded, enter:

   ~]# lsmod |grep rdma_rxe
     rdma_rxe              111129  0
     ip6_udp_tunnel         12755  1 rdma_rxe
     udp_tunnel             14423  1 rdma_rxe
     ib_core               236827  15 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_rxe,rdma_ucm,ib_ipoib,ib_isert
   

3. Before adding a new RXE device over an Ethernet interface, the corresponding interface should be opened and has a valid IP address assigned. To add a new RXE device, for example igb_1:

   ~]# rxe_cfg add igb_1

   ~]# rxe_cfg status
     Name        Link  Driver   Speed   NMTU  IPv4_addr  RDEV  RMTU
     igb_1       yes   igb                               rxe0  1024  (3)
     mlx4_1      no    mlx4_en
     mlx4_2      no    mlx4_en

   The rxe0 in the RDEV column indicates that rxe is enabled for the igb_1 device.
4. To verify the status of an RXE device, use the ibv_devices command:

   ~]# ibv_devices
     device                 node GUID
     ------              ----------------
     mlx4_0              0002c90300b3cff0
     rxe0                a2369ffffe018294

   Alternatively, enter the ibstat for a detailed status:

   ~]# ibstat rxe0
   CA 'rxe0'
       CA type:
       Number of ports: 1
       Firmware version:
       Hardware version:
       Node GUID: 0xa2369ffffe018294
       System image GUID: 0x0000000000000000
       Port 1:
           State: Active
           Physical state: LinkUp
           Rate: 2.5
           Base lid: 0
           LMC: 0
           SM lid: 0
           Capability mask: 0x00890000
           Port GUID: 0xa2369ffffe018294
           Link layer: Ethernet