Kdump over NFS fails to generate vmcore on a system with Intel network card with IDPF driver

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9.4
  • Intel network card with IDPF driver

Issue

Kdump over NFS fails to generate vmcore on a system with one or more Intel network card(s) that use the idpf driver.

The network manager times out waiting for a non-existent device; after that it requests a DHCP address, which fails with no response. Eventually, kdump is disconnected and fails.

[ 6.984406] systemd[1]: Starting nm-wait-online-initrd.service...
[ 9.433201] idpf 0000:05:00.0: Device HW Reset initiated
[ 10.478322] 4xxx 0000:6b:00.0: Failed to power up the device
[ 10.484005] 4xxx 0000:6b:00.0: Failed to initialize device
[ 10.499766] 4xxx 0000:6b:00.0: Resetting device qat_dev0
[ 10.612515] 4xxx 0000:6b:00.0: probe with driver 4xxx failed with error -14
[[0m[0;31m* [0m] A start job is running for nm-wait-…ine-initrd.service (5s / no limit)
[ 11.626921] 4xxx 0000:e8:00.0: Failed to power up the device
[ 11.632606] 4xxx 0000:e8:00.0: Failed to initialize device
[ 11.649413] 4xxx 0000:e8:00.0: Resetting device qat_dev0
[ 9.701809] rngd[494]: [jitter]: Enabling JITTER rng support
[ 9.709045] rngd[494]: [jitter]: Initialized
[ 11.757398] 4xxx 0000:e8:00.0: probe with driver 4xxx failed with error -14

After that NM eventually times out waiting for a device:

[ 66.584985] NetworkManager[493]: <warn> [1734752025.5989] settings: startup-complete: profile "Wired connection 1" (901471d3-9a88-3504-83c2-693ab17abadb) was waiting for non-existing device (with timeout "connection.wait-device-timeout=60000")
...
[ 72.684326] idpf 0000:05:00.0: Transaction timed-out (op 1, 60000ms)
[ 72.721325] idpf 0000:05:00.0: Device HW Reset initiated
[ 71.089973] kdump[649]: Waiting for network to be ready (4s / 10min)
...
[ 72.109371] systemd-udevd[657]: Network interface NamePolicy= disabled on kernel command line.
[ 72.119415] NetworkManager[493]: <info> [1734752031.1237] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
[ 72.132189] NetworkManager[493]: <info> [1734752031.1279] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
[ 73.104718] kdump[666]: Waiting for network to be ready (6s / 10min)
[ 73.212226] NetworkManager[493]: <info> [1734752032.2201] device (eth0): carrier: link connected
[ 73.222162] dhclient[668]: DHCPDISCOVER on eth0 to
255.255.255.255
port 67 interval 7 (xid=0xb41e662)

Resolution

Red Hat Enterprise Linux 9.4.z

  • The issue has been resolved in kernel version kernel-5.14.0-427.127.1.el9_4

Red Hat Enterprise Linux 9.6

Red Hat Enterprise Linux 10

Root Cause

The idpf driver didn't call netif_set_real_num_tx_queues() during initial boot. Changing the number of channels triggered an idpf soft reset, which then invoked the function. After that, TX packets started flowing, and DHCP and ARP pings worked.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments