Tools that poll for networking information are slow (ie: ifconfig, ip, net-snmp...)

Solution In Progress - Updated -

Environment

Red Hat Enterprise Linux 6.6 (kernel 2.6.32-504.el6)
SFP+ capable Intel network cards

Issue

Running commands to get networking related information takes long time ( >1 second) to provide the output.
This is particularly evident in case of large number of interfaces, and it can be observed with the following:

# time ifconfig |grep -i "encap.*Ethernet" | wc -l
37

real    0m1.244s
user    0m0.001s
sys 0m0.008s

Resolution

Soluiton: The issue was fixed in Red Hat Enterprise Linux 6.6 kernel 2.6.32-504.23.4.el6, so update to this kernel.

Workaround 1: The bug is triggered by loading the driver when one or more network cards have no SFP+ module mounted, so, if you have related hardware (SFP+ modules) you can just insert at least one SFP+ module in each ixgbe NIC.

Workaround 2: Blacklist buggy [ixgbe] driver from loading.

Root Cause

The observed behaviour is due to a bug in the ixgbe driver.
The bug is triggered by loading the driver when 1 or more cards have no SFP+ module mounted. In this case the driver would not recognize the card as SFP+ capable triggering the call to identify_sfp in ixgbe_get_settings in order to verify the correctness of the identification.
If the SFP+ module is not present, the call would wait for the I2C reads timeout, causing the observed delay.

Diagnostic Steps

You can compare the [ixgbe] module code with the upstream patch and code of RHEL kernel released to be sure the patch is incuded:

Upstream ixgbe Patch

kernel 2.6.32-504.23.4.el6 ixgbe_ethtool.c

kernel 2.6.32-504.23.4.el6 ixgbe_main.c

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments