Broadcom online firmware updates not working with tg3 driver version 3.137.

Solution Verified - Updated -

Environment

  • Red Hat Enterpirse Linux (RHEL) 6.6
    • kernel-2.6.32-504.el6 - kernel-2.6.32-504.8.1.el6
    • tg3 version 3.137

Issue

Broadcom online firmware updates not working with tg3 driver version 3.137 which is included as part of Red Hat Enterprise Linux 6.6 kernel-2.6.32-504.el6.

Resolution

  • Update to kernel-2.6.32-504.12.2.el6 released via RHSA-2015:0674 (or later)

  • Update to kernel-2.6.32-573.el6 re;eased via RHSA-2015:1272 (or later)

    • If a user attempted to apply an NVRAM firmware update when running the tg3 module provided with Red Hat Enterprise Linux 6.6 kernels, the update could fail. As a consequence, the Network Interface Card (NIC) could stay in an unusable state and this could prevent the entire system from booting. The tg3 module has been updated to correctly apply firmware updates. (BZ#1182903)

  • While updating kernel RPM is recommended solution, some other possible options for doing firmware updates are:

    • Boot from an alternate (ie. older) kernel that provides older tg3 driver version
    • Use hardware manufacture's offline firmware update method.
    • Use hardware manufacture's provided tg3 driver that has already addressed timeout issue.
      • Hewlett Packard Support Communication c04569875

Root Cause

The tg3 driver decrease of the NVRAM command execution timeout value from 100ms to 1ms in the following patches:

Upstream has since changed this timeout from 1ms to 50ms as the 1ms timeout was not long enough for write operations. However, the tg3 driver in RHEL 6.6 GA does not contain the following upstream patch until kernel-2.6.32-504.12.2.el6 (or later).

Diagnostic Steps

Looking at kernel RPM changelog can see where the initial change was done in BZ 1054558 and then corrected in BZ 1182903 / BZ 1176230.

$ rpm -qp --changelog kernel-2.6.32-504.12.2.el6.x86_64.rpm | grep -e 'tg3: Override clock' -e 'tg3: Change.*timeout' -e '^\*' | grep -B1 tg3
* Thu Jan 22 2015 Radomir Vrbovsky <rvrbovsk@redhat.com> [2.6.32-504.10.1.el6]
- [netdrv] tg3: Change nvram command timeout value to 50ms (Ivan Vecera) [1182903 1176230]
--
* Wed Jun 11 2014 Rafael Aquini <aquini@redhat.com> [2.6.32-479.el6]
- [netdrv] tg3: Override clock, link aware and link idle mode during NVRAM dump (Ivan Vecera) [1054558]

Here is how can check the kernel version, and more crucially the tg3 driver currently in use.

# uname -r
2.6.32-504.el6.x86_64

# modinfo tg3 | grep ^version
version:        3.137

Issue has been reported with HP online firmware CP021848.scexe, CP020090.scexe, and CP023977.scexe. Here is sample output of one of the firmware update failures that may leave the system unusable after reboot.

# ./CP020090.scexe
HP Gigabit Server Adapter Firmware Upgrade Utility

Found HP Ethernet 1Gb 4-port 331FLR Adapter MAC: D89D6717EE58
Update Boot Code 1.24 to 1.33 -  y/n/q/c (y):y
Update PXE 15.0.12 to 15.4.2 -  y/n/q/c (y):y
Update NCSI 1.0.60 to 1.1.15 -  y/n/q/c (y):y

The Firmware Upgrade will take several minutes.  Please be patient.

Selecting HP Ethernet 1Gb 4-port 331FLR Adapter MAC: D89D6717EE58
Upgrading Boot Code 1.24 to 1.33
Broadcom Network Controller Firmware Upgrade Utility for Linux v2.2.5
Initializing...

Select NIC NetXtreme BCM5719 Gigabit Ethernet PCIe rev 01 (eth0)...
Region           Address Range     Content  Checksum Status
---------------- ----------------- -------- -------- -------
Bootstrap        00000000-00000013 07BAB558 07BAB558 Ok
Directory        00000014-00000073 00000000 00000000 Ok
Manufacture      00000074-000000FF 37ADF754 37ADF754 Ok
Manufacture II   00000200-0000028B 446FDC1D 446FDC1D Ok
Boot Code        0000028C-00001BAF B0271227 B0271227 Ok
2nd Boot Code    00001BB8-000048E7 A992EA48 A992EA48 Ok
Validating manufacturing information ...OK
Running NVRAM gap cleanning routine...
Reading PXE data from 0x00004490...
Reading APE Firmware data from 0x00020DC8...
Reading Extended VPD data from 0x0002936C...
Writing PXE data to 000048E8...
Failed to write NVRAM!
Please use 'restorenvram' command to restore the original NVRAM content!


UnInitializing...
Program Exit Code: (18)
ERROR: Upgrade of HP Ethernet 1Gb 4-port 331FLR Adapter MAC: D89D6717EE58
failed.
Upgrading PXE 15.0.12 to 15.4.2
Broadcom Network Controller Firmware Upgrade Utility for Linux v2.2.5
Initializing...

Select NIC NetXtreme BCM5719 Gigabit Ethernet PCIe rev 01 (eth0)...
Updating PCI ROM (type 0) header with Vendor ID = 0x14e4, Device ID = 0x1657
Updating PXE UNDI16 (length = 120948 bytes ) ...
Programming 120948 bytes to eeprom offset 0x00029690
Failed to write NVRAM!
Please use 'restorenvram' command to restore the original NVRAM content!


UnInitializing...
Program Exit Code: (18)
ERROR: Upgrade of HP Ethernet 1Gb 4-port 331FLR Adapter MAC: D89D6717EE58
failed.
Upgrading NCSI 1.0.60 to 1.1.15
Broadcom Network Controller Firmware Upgrade Utility for Linux v2.2.5
Initializing...

Select NIC NetXtreme BCM5719 Gigabit Ethernet PCIe rev 01 (eth0)...
RSA Signature verified.
Programming 35056 bytes to eeprom offset 0x00029690
Failed to write NVRAM!
Please use 'restorenvram' command to restore the original NVRAM content!

APE Firmware programmed successfully.


UnInitializing...
Program Exit Code: (18)
ERROR: Upgrade of HP Ethernet 1Gb 4-port 331FLR Adapter MAC: D89D6717EE58
failed.

NIC firmware update completed with errors.

This issue is specific to the 3.137 version of the tg3 network driver. Please be aware of the following article regarding a different issue with online firmware updates with Hewlett-Packard's online firmware updates for Broadcom NICs.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments