How to install support for Mellanox Infiniband hardware on RHEL6

Solution Verified - Updated -

Environment

Red Hat Enterprise Linux 6
Mellanox Infiniband hardware

Issue

Mellanox Infiniband hardware support in RHEL6 should be properly installed before use. Hardware drivers and Infiniband-related packages are not installed by default.

Resolution

According to Does Red Hat Enterprise Linux support FDR Infiniband devices? article, QDR (Quad Data Rate) Mellanox hardware is supported in Red Hat Enterprise Linux 6 and FDR (Fourteen Data Rate) hardware is supported since Red Hat Enterprise Linux version 6 update 3.

The following should be made to install Infiniband support for Mellanox hardware on Red Hat Enterprise Linux 6:

1) Install the packages required and reboot:

[root@serv]# yum groupinstall "Infiniband Support"
[root@serv]# yum install infiniband-diags perftest qperf opensm
[root@serv]# chkconfig rdma on
[root@serv]# chkconfig opensm on
[root@serv]# shutdown -r now

2) Check that IB ports are available (your output should differ):

[root@serv]# for i in `ls /sys/class/infiniband/*/ports/*/state`; do echo $i; cat $i; done
/sys/class/infiniband/mlx4_0/ports/1/state
4: ACTIVE
/sys/class/infiniband/mlx4_0/ports/2/state
1: DOWN

3) Get current IB HCA state and topology (your output should differ):

(shows IB host nodes in topology)

[root@serv]# ibhosts

(reports link info for all links in the fabric)

[root@serv]# iblinkinfo
Switch 0x0002c9020041e0b8 Infiniscale-IV Mellanox Technologies:
2 1[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" ( )
...skip...
2 31[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 16 1[ ] "serv-2 HCA-1" ( )
2 32[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 17 1[ ] "serv HCA-1" ( )

4) (optional) Run some testing commands to ensure connectivity (your output and commands arguments should differ):

[root@serv]# ibtracert 16 17
[root@serv]# ibping -S -v
[root@serv]# ibping -c 30 16

5) (optional) Disable MSI if you have 'bnx2' network cards:

[root@serv]# cat /etc/modprobe.d/bnx2-msi.conf
options bnx2 disable_msi=1

Please see additional details in How do I disable MSI for the Broadcom bnx2 kernel module on Red Hat Enterprise Linux 6 ?

6) Further configuration steps
Please, see the article How can I get started using Infiniband on Red Hat Enterprise Linux ? after you have installed drivers and ensured Infiniband connectivity.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments