Chapter 6. Testing InfiniBand networks

This section provides procedures how to test InfiniBand networks.

6.1. Testing early InfiniBand RDMA operations

This section describes how to test InfiniBand remote direct memory access (RDMA) operations.

Note

This section applies only to InfiniBand devices. If you use iWARP or RoCE/IBoE devices, which are IP-based, see:

Prerequisites

  • RDMA is configured.
  • The libibverbs-utils and infiniband-diags packages are installed.

Procedure

  1. List the available InfiniBand devices: 

    # ibv_devices
    device                 node GUID
    ------              ----------------
    mlx4_0              0002c903003178f0
    mlx4_1              f4521403007bcba0

  2. Display the information for a specific InfiniBand device. For example, to display the information of the mlx4_1 device, enter: 

    # ibv_devinfo -d mlx4_1
hca_id: mlx4_1
     transport:                  InfiniBand (0)
     fw_ver:                     2.30.8000
     node_guid:                  f452:1403:007b:cba0
     sys_image_guid:             f452:1403:007b:cba3
     vendor_id:                  0x02c9
     vendor_part_id:             4099
     hw_ver:                     0x0
     board_id:                   MT_1090120019
     phys_port_cnt:              2
          port:   1
                state:              PORT_ACTIVE (4)
                max_mtu:            4096 (5)
                active_mtu:         2048 (4)
                sm_lid:             2
                port_lid:           2
                port_lmc:           0x01
                link_layer:         InfiniBand

          port:   2
                state:              PORT_ACTIVE (4)
                max_mtu:            4096 (5)
                active_mtu:         4096 (5)
                sm_lid:             0
                port_lid:           0
                port_lmc:           0x00
                link_layer:         Ethernet

  3. Display the basic status of an InfiniBand device. For example, to display the status of the mlx4_1 device, enter: 

    # ibstat mlx4_1
CA 'mlx4_1'
     CA type: MT4099
     Number of ports: 2
     Firmware version: 2.30.8000
     Hardware version: 0
     Node GUID: 0xf4521403007bcba0
     System image GUID: 0xf4521403007bcba3
     Port 1:
           State: Active
           Physical state: LinkUp
           Rate: 56
           Base lid: 2
           LMC: 1
           SM lid: 2
           Capability mask: 0x0251486a
           Port GUID: 0xf4521403007bcba1
           Link layer: InfiniBand
     Port 2:
           State: Active
           Physical state: LinkUp
           Rate: 40
           Base lid: 0
           LMC: 0
           SM lid: 0
           Capability mask: 0x04010000
           Port GUID: 0xf65214fffe7bcba2
           Link layer: Ethernet

  4. Use the ibping utility to ping from a client to a server using InfiniBand:

    1. On the host that acts as a server, start ibping in server mode: 

      # ibping -S -C mlx4_1 -P 1

      This command uses the following parameters:

      • -S: Enables the server mode.
      • -C InfiniBand_CA_name: Set’s the CA name to use.
      • -P port_number: Sets the port number to use, if the InfiniBand provides multiple ports.

    2. On the host that acts as client, use ibping as follows: 

      # ibping -c 50 -C mlx4_0 -P 1 -L 2
      • -c number: Sends these number of packets to the server.
      • -C InfiniBand_CA_name: Set’s the CA name to use.
      • -P port_number: Sets the port number to use, if the InfiniBand provides multiple ports.
      • -L port_LID: Sets the Local Identifier (LID) to use.

Additional resources

  • For further details about ibping parameters, see the ibping(8) man page.

6.2. Testing an IPoIB using the ping utility

After you configured IPoIB, use the ping utility to send ICMP packets to test the IPoIB connection.

Prerequisites

  • The two RDMA hosts are connected in the same InfiniBand fabric with RDMA ports.
  • The IPoIB interfaces in both hosts are configured with IP addresses within the same subnet.

Procedure

  1. Use the ping utility to send ICMP packets to the remote host’s InfiniBand adapter: 

    # ping -c5 192.0.2.1

    This command sends five ICMP packets to the IP address 192.0.2.1.

6.3. Testing an RDMA network using qperf after IPoIB is configured

This procedure describes examples how to display the InfiniBand adapter configuration and measure the bandwidth and latency between two hosts using the qperf utility.

Prerequisites

  • The qperf package is installed on both hosts.
  • IPoIB is configured on both hosts.

Procedure

  1. Start qperf on one of the hosts without any options to act as a server: 

    # qperf

  2. Use the following commands on the client. The commands use port 1 of the mlx4_0 host channel adapter in the client to connect to IP address 192.0.2.1 assigned to the InfiniBand adapter in the server.

    1. To display the configuration, enter: 

      qperf -v -i mlx4_0:1 192.0.2.1 conf
-------------------------
conf:
    loc_node   =  rdma-dev-01.lab.bos.redhat.com
    loc_cpu    =  12 Cores: Mixed CPUs
    loc_os     =  Linux 4.18.0-187.el8.x86_64
    loc_qperf  =  0.4.11
    rem_node   =  rdma-dev-00.lab.bos.redhat.com
    rem_cpu    =  12 Cores: Mixed CPUs
    rem_os     =  Linux 4.18.0-187.el8.x86_64
    rem_qperf  =  0.4.11
-------------------------

    2. To display the Reliable Connection (RC) streaming two-way bandwidth, enter: 

      # qperf -v -i mlx4_0:1 192.0.2.1 rc_bi_bw
-------------------------
rc_bi_bw:
    bw             =  10.7 GB/sec
    msg_rate       =   163 K/sec
    loc_id         =  mlx4_0
    rem_id         =  mlx4_0:1
    loc_cpus_used  =    65 % cpus
    rem_cpus_used  =    62 % cpus
-------------------------

    3. To display the RC streaming one-way bandwidth, enter: 

      # qperf -v -i mlx4_0:1 192.0.2.1 rc_bw
-------------------------
rc_bw:
    bw              =  6.19 GB/sec
    msg_rate        =  94.4 K/sec
    loc_id          =  mlx4_0
    rem_id          =  mlx4_0:1
    send_cost       =  63.5 ms/GB
    recv_cost       =    63 ms/GB
    send_cpus_used  =  39.5 % cpus
    recv_cpus_used  =    39 % cpus
-------------------------

Additional resources

  • For further details about qperf, see the qperf(1) man page.