Mellanox MT26448 ConnectX goes offline when using ethtool -G in RHEL 6.4

Issue

• Changing the ring buffer setting using ethtool -G causes the NIC to go offline in RHEL 6.4, kernel-2.6.32-358.6.2.

• HOST-A (HP DL360 G6) has eth4.13, eth4.14,eth4.202, and eth4.203 on the Mellanox ConnectX-2 card. As soon as nictune runs, the NIC goes offline.

• HOST-B (HP DL360 G6) has eth2 and eth3 in bond0 (VLAN 164) on a Broadcom, and then eth4.202, eth4.204, eth5.203, eth5.205, and eth5.198 on the Mellanox ConnectX-2 card. Same issue with nictune above.

• HOST-C (HP DL360p Gen8) has eth2 and eth3 in bond0 (VLAN 160) on the Mellanox ConnectX-3 and works OK

• So either, VLAN tagging of a NIC directly (not a bond) doesn't work right, or much more likely the ConnectX-2 cards are broken in the new Red Hat driver or firmware.

• The issue arises when we run 'ethtool -G'. We normally run that on boot in order to set the ring buffer of each NIC driver to the max settings for RX and TX. We were able to work around this issue by disabling the ring buffer tuning; however, we expect during high market load we will run into NIC driver overruns using the default values.