12.12. Testing an RDMA network after IPoIB is configured

Once IPoIB is configured, it is possible to use IP addresses to specify RDMA devices. Due to the ubiquitous nature of using IP addresses and host names to specify machines, most RDMA applications use this as their preferred, or in some cases only, way of specifying remote machines or local devices to connect to.
To test the functionality of the IPoIB layer, it is possible to use any standard IP network test tool and provide the IP address of the IPoIB devices to be tested. For example, the ping command between the IP addresses of the IPoIB devices should now work.
There are two different RDMA performance testing packages included with Red Hat Enterprise Linux, qperf and perftest. Either of these may be used to further test the performance of an RDMA network.
However, when using any of the applications that are part of the perftest package, or using the qperf application, there is a special note on address resolution. Even though the remote host is specified using an IP address or host name of the IPoIB device, it is allowed for the test application to actually connect through a different RDMA interface. The reason for this is because the process of converting from the host name or IP address to an RDMA address allows any valid RDMA address pair between the two machines to be used. If there are multiple ways for the client to connect to the server, then the programs might choose to use a different path if there is a problem with the path specified. For example, if there are two ports on each machine connected to the same InfiniBand subnet, and an IP address for the second port on each machine is given, it is likely that the program will find the first port on each machine is a valid connection method and use them instead. In this case, command-line options to any of the perftest programs can be used to tell them which card and port to bind to, as was done with ibping in Section 12.7, “Testing Early InfiniBand RDMA operation”, in order to ensure that testing occurs over the specific ports required to be tested. For qperf, the method of binding to ports is slightly different. The qperf program operates as a server on one machine, listening on all devices (including non-RDMA devices). The client may connect to qperf using any valid IP address or host name for the server. Qperf will first attempt to open a data connection and run the requested test(s) over the IP address or host name given on the client command line, but if there is any problem using that address, qperf will fall back to attempting to run the test on any valid path between the client and server. For this reason, to force qperf to test over a specific link, use the -loc_id and -rem_id options to the qperf client in order to force the test to run on a specific link.