13.3. Configuring Soft-RoCE
Prerequisites
rxe_cfg
script. To view options for rxe_cfg
, enter rxe_cfg help
.
Procedure 13.2. Configuring Soft-RoCE
- As the
root
user, enter the following command to display the current configuration status of RXE:~]#
rxe_cfg
rdma_rxe module not loaded Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1yes
igb mlx4_1 no mlx4_en mlx4_2 no mlx4_en - To load the RXE kernel module and start RXE, enter as
root
:~]#
Optionally, to verify that the RXE kernel module is loaded, enter:rxe_cfg start
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1yes
igb mlx4_1 no mlx4_en mlx4_2 no mlx4_en~]#
lsmod |grep rdma_rxe
rdma_rxe 111129 0 ip6_udp_tunnel 12755 1 rdma_rxe udp_tunnel 14423 1 rdma_rxe ib_core 236827 15 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx4_ib,ib_srp,ib_ucm,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_rxe,rdma_ucm,ib_ipoib,ib_isert - Before adding a new RXE device over an Ethernet interface, the corresponding interface should be opened and has a valid IP address assigned. To add a new RXE device, for example igb_1:
~]#
rxe_cfg add igb_1
~]#
The rxe0 in the RDEV column indicates that rxe is enabled for the igb_1 device.rxe_cfg status
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU igb_1 yes igb rxe0 1024 (3) mlx4_1 no mlx4_en mlx4_2 no mlx4_en - To verify the status of an RXE device, use the
ibv_devices
command:~]#
Alternatively, enter theibv_devices
device node GUID ------ ---------------- mlx4_0 0002c90300b3cff0 rxe0 a2369ffffe018294ibstat
for a detailed status:~]#
ibstat
rxe0 CA 'rxe0' CA type: Number of ports: 1 Firmware version: Hardware version: Node GUID: 0xa2369ffffe018294 System image GUID: 0x0000000000000000 Port 1: State: Active Physical state: LinkUp Rate: 2.5 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x00890000 Port GUID: 0xa2369ffffe018294 Link layer: Ethernet
Removing an RXE device
~]# rxe_cfg remove igb_1
Verifying Connectivity of an RXE device
Example 13.1. Verifying Connectivity of an RXE device on the Server Side
~]$ ibv_rc_pingpong -d rxe0 -g 0
local address: LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
remote address: LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
8192000 bytes in 0.05 seconds = 1244.06 Mbit/sec
1000 iters in 0.05 seconds = 52.68 usec/iter
Example 13.2. Verifying Connectivity of an RXE device on the Client Side
~]$ ibv_rc_pingpong -d rxe0 -g 0 172.31.40.4
local address: LID 0x0000, QPN 0x000011, PSN 0x4bf206, GID fe80::290:faff:fe29:470a
remote address: LID 0x0000, QPN 0x000012, PSN 0xe2965f, GID fe80::290:faff:fe29:486a
8192000 bytes in 0.05 seconds = 1245.72 Mbit/sec
1000 iters in 0.05 seconds = 52.61 usec/iter