Why does the Broadcom NetXtreme 5709 NIC stop receiving packets intermittently on RHEL 5.3 and newer?
Red Hat Enterprise Linux 5.3 to 5.5
Network Interface Cards (NIC) using the bnx2 driver including:
- Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet
- In certain situations under heavy loads, the network interface card can stop accepting packets from remote devices.
- This problem has been reported on Red Hat Enterprise Linux 5.3 (RHEL 5.3) and newer when using a Broadcom NetXtreme 5709 network interface card.
- Red Hat has released
kernel-2.6.18-194.3.1.el5which will address this issue in RHEL 5. It can be downloaded from the following link:
- in certain circumstances, under heavy load, certain network interface cards using the bnx2 driver and configured to use MSI-X(extended MSI), could stop processing interrupts and then network connectivity would cease. Bugzilla 587799
If upgrading the kernel is not an option, review the following workarounds
- Disable MSI-X in the bnx2 driver. To do this, add the following line to /etc/modprobe.conf
options bnx2 disable_msi=1
Disable MSI(message signalled interrupt) completely by booting with the pci=nomsi boot parameter. Obviously, this will disable MSI on all devices that are able to utilize it.
Note: MSI-X increases network performance, so disabling it means that the performance will return to the level available before MSI-X was introduced.
Disable C-States in BIOS. Refer to the vendor system documentation in order to learn how to do this.
- The kernel gets out of sync with interrupts generated by the network interface card which results in an inability to process interrupts, causing packets to be dropped and ultimately, lost connectivity.
- When this situation occurs, the rx_fw_discards counter will keep increasing as remote devices unsuccessfully attempt to communicate with the system via the NIC.
- It has been reported that under certain heavy traffic conditions in MSI-X mode, the bnx2 driver can lose an MSI-X vector causing all packets in the associated rx/tx ring pair to be dropped. The problem is caused by the chip dropping the write to unmask the MSI-X vector by the kernel (when migrating the IRQ for example).This can be prevented by increasing the GRC timeout value for these register read and write operations.
- The upstream patch resolving this issue is available here:
Commit id : c441b8d2cb2194b05550a558d6d95d8944e56a84
- The kernel gets out of sync with regard to the interrupts generated by the network interface card which prevents the reception of packets on this network device which results in no processing of interrupts, dropped packets and ultimately, lost connectivity.
- When this situation occurs, the rx_fw_discards counter displayed by the ethtool utility will keep increasing in value as remote devices unsuccessfully attempt to communicate with the system via the NIC.
- It should be noted that packets are occasionally dropped by the NIC as part of normal operation which causes rx_fw_discards to increment, but this does not necessarily indicate the issue in question has manifested.
The keys to determining that this specific problem has occurred are:
- Confirm that all packets sent to the NIC are dropped by repeatedly using this command:
# ethtool -S eth0 | grep rx_fw_discards
(Replace "eth0" with the interface that appears to be having trouble receiving)
Each time this command is executed, the value returned should increase from the previous run as a result of remote devices attempting to communicate with the NIC in question. The numbers should increase similar to this:
rx_fw_discards: 53843 rx_fw_discards: 55467 rx_fw_discards: 57071 rx_fw_discards: 58791 rx_fw_discards: 60596 rx_fw_discards: 62481 rx_fw_discards: 64285 rx_fw_discards: 66069
- Confirm that the number of interrupts processed does not increase on the IRQs assigned to the NIC by repeatedly using this command:
# grep eth0 /proc/interrupts
(Modify "eth0" with the name of the interface where trouble is suspected.)
The command should be run while remote devices are attempting to transmit to the failing system. Normally, each counter for the interrupts listed for that interface (e.g. eth0) should increase as packets are received from remote devices. In this situation being described here, the interrupt counter(s) should stop incrementing. In severe cases, the counters for all interrupts can remain constant and then the interface will receive no packets from any remote device.
- Typically there is no syslog or dmesg output to indicate the issue has occurred.
- Red Hat Enterprise Linux
- Learn more
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
This solved our NFS-problem with a RHEL5 and NetApp NFS storage connection.
After adding the above suggestion (options bnx2 disable_msi=1) in /etc/modprobe.conf , still we are experiencing the packet drop.
Would we need to reboot the machine to work the changes after adding /etc/modprobe.conf?
Is this solution stil applicable on RHEL 6.x and RHEL 7.y?