Chapter 7. Testing InfiniBand networks

7.1. Testing early InfiniBand RDMA operations

InfiniBand provides low latency and high performance for Remote Direct Memory Access (RDMA).

Note

Apart from InfiniBand, if you use IP-based devices such as Internet Wide-area Remote Protocol(iWARP) or RDMA over Converged Ethernet (RoCE) or InfiniBand over Ethernet (IBoE) devices, see:

Prerequisites

  • You’ve configured the rdma service.
  • You’ve installed the libibverbs-utils and infiniband-diags packages.

Procedure

  1. List the available InfiniBand devices: 

    # ibv_devices

    device                 node GUID
    ------              ----------------
    mlx4_0              0002c903003178f0
    mlx4_1              f4521403007bcba0

  2. Display the information of the mlx4_1 device: 

    # ibv_devinfo -d mlx4_1

hca_id: mlx4_1
     transport:                  InfiniBand (0)
     fw_ver:                     2.30.8000
     node_guid:                  f452:1403:007b:cba0
     sys_image_guid:             f452:1403:007b:cba3
     vendor_id:                  0x02c9
     vendor_part_id:             4099
     hw_ver:                     0x0
     board_id:                   MT_1090120019
     phys_port_cnt:              2
          port:   1
                state:              PORT_ACTIVE (4)
                max_mtu:            4096 (5)
                active_mtu:         2048 (4)
                sm_lid:             2
                port_lid:           2
                port_lmc:           0x01
                link_layer:         InfiniBand

          port:   2
                state:              PORT_ACTIVE (4)
                max_mtu:            4096 (5)
                active_mtu:         4096 (5)
                sm_lid:             0
                port_lid:           0
                port_lmc:           0x00
                link_layer:         Ethernet

  3. Display the status of the mlx4_1 device: 

    # ibstat mlx4_1

CA 'mlx4_1'
     CA type: MT4099
     Number of ports: 2
     Firmware version: 2.30.8000
     Hardware version: 0
     Node GUID: 0xf4521403007bcba0
     System image GUID: 0xf4521403007bcba3
     Port 1:
           State: Active
           Physical state: LinkUp
           Rate: 56
           Base lid: 2
           LMC: 1
           SM lid: 2
           Capability mask: 0x0251486a
           Port GUID: 0xf4521403007bcba1
           Link layer: InfiniBand
     Port 2:
           State: Active
           Physical state: LinkUp
           Rate: 40
           Base lid: 0
           LMC: 0
           SM lid: 0
           Capability mask: 0x04010000
           Port GUID: 0xf65214fffe7bcba2
           Link layer: Ethernet

  4. The ibping utility pings an InfiniBand address and runs as a client/server by configuring the parameters.

    1. Start server mode -S on port number -P with -C InfiniBand certificate authority (CA) name on the host: 

      # ibping -S -C mlx4_1 -P 1

    2. Start client mode, send some packets -c on port number -P using -C InfiniBand certificate authority (CA) name with -L Local Identifier (LID) on the host: 

      # ibping -c 50 -C mlx4_0 -P 1 -L 2

Additional resources

  • ibping(8) man page

7.2. Testing an IPoIB using the ping utility

After you configured IP over InfiniBand (IPoIB), use the ping utility to send ICMP packets to test the IPoIB connection.

Prerequisites

  • The two RDMA hosts are connected in the same InfiniBand fabric with RDMA ports
  • The IPoIB interfaces in both hosts are configured with IP addresses within the same subnet

Procedure

  • Use the ping utility to send five ICMP packets to the remote host’s InfiniBand adapter: 

    # ping -c5 192.0.2.1

7.3. Testing an RDMA network using iperf3 after IPoIB is configured

In the following example, the large buffer size is used to perform a 60 seconds test to measure maximum throughput and fully use the bandwidth and latency between two hosts using the iperf3 utility.

Prerequisites

  • You’ve configured IPoIB on both hosts.

Procedure

  1. To run iperf3 as a server on a system, define a time interval to provide periodic bandwidth updates -i to listen as a server -s that waits for the response of the client connection: 

    # iperf3 -i 5 -s

  2. To run iperf3 as a client on another system, define a time interval to provide periodic bandwidth updates -i to connect to the listening server -c of IP address 192.168.2.2 with -t time in seconds: 

    # iperf3 -i 5 -t 60 -c 192.168.2.2

  3. Use the following commands:

    1. Display test results on the system that acts as a server: 

      # iperf3 -i 10 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.3, port 22216
[5] local 192.168.2.2 port 5201 connected to 192.168.2.3 port 22218
[ID] Interval           Transfer     Bandwidth
[5]   0.00-10.00  sec  17.5 GBytes  15.0 Gbits/sec
[5]  10.00-20.00  sec  17.6 GBytes  15.2 Gbits/sec
[5]  20.00-30.00  sec  18.4 GBytes  15.8 Gbits/sec
[5]  30.00-40.00  sec  18.0 GBytes  15.5 Gbits/sec
[5]  40.00-50.00  sec  17.5 GBytes  15.1 Gbits/sec
[5]  50.00-60.00  sec  18.1 GBytes  15.5 Gbits/sec
[5]  60.00-60.04  sec  82.2 MBytes  17.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ID] Interval           Transfer     Bandwidth
[5]   0.00-60.04  sec  0.00 Bytes    0.00 bits/sec  sender
[5]   0.00-60.04  sec   107 GBytes  15.3 Gbits/sec  receiver

    2. Display test results on the system that acts as a client: 

      # iperf3 -i 1 -t 60 -c 192.168.2.2

Connecting to host 192.168.2.2, port 5201
[4] local 192.168.2.3 port 22218 connected to 192.168.2.2 port 5201
[ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[4]   0.00-10.00  sec  17.6 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  10.00-20.00  sec  17.6 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  20.00-30.00  sec  18.4 GBytes  15.8 Gbits/sec    0   6.01 MBytes
[4]  30.00-40.00  sec  18.0 GBytes  15.5 Gbits/sec    0   6.01 MBytes
[4]  40.00-50.00  sec  17.5 GBytes  15.1 Gbits/sec    0   6.01 MBytes
[4]  50.00-60.00  sec  18.1 GBytes  15.5 Gbits/sec    0   6.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ID] Interval           Transfer     Bandwidth       Retr
[4]   0.00-60.00  sec   107 GBytes  15.4 Gbits/sec    0   sender
[4]   0.00-60.00  sec   107 GBytes  15.4 Gbits/sec        receiver

Additional resources

  • iperf3 man page