After upgrading the kernel, ice/iavf driver and SR-IOV workloads stop working
Issue
- After upgrading the kernel, the SR-IOV workloads stop functioning and hit the following error in the guest:
iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
- Network flows are interrupted, the error keeps repeating.
- Panic occurred with below logs:
[ 2274.923162] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2279.135890] iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 2279.156168] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2285.062016] BUG: unable to handle kernel paging request at 000054860000343f
[ 2285.063051] PGD 0
[ 2285.063419] Oops: 0002 [#1] SMP NOPTI
[ 2285.063964] CPU: 3 PID: 4861 Comm: kworker/u8:1 Kdump: loaded Tainted: G W --------- - - 4.18.0-425.3.1.el8.x86_64 #1
[ 2285.065586] Hardware name: Red Hat OpenStack Compute, BIOS 1.14.0-1.module+el8.4.0+8855+a9e237a9 04/01/2014
[ 2285.066863] Workqueue: iavf iavf_reset_task [iavf]
[ 2285.067557] RIP: 0010:__page_frag_cache_drain+0x5/0x30
[ 2285.070735] RSP: 0018:ff4c815a0624fdf0 EFLAGS: 00010292
[ 2285.071474] RAX: ffffffff9d5b9740 RBX: ff20f48b074b0000 RCX: 0000000000000002
[ 2285.072441] RDX: 00000000af000000 RSI: 0000000000000000 RDI: 000054860000340b
[ 2285.073412] RBP: ff20f48b0a5d1c00 R08: 0000000000000022 R09: ff20f48b004027e0
[ 2285.074384] R10: 587829f300000000 R11: 0000000000000020 R12: 0000000000000000
[ 2285.075357] R13: 0000000000001000 R14: ff20f48b1dbc0d10 R15: 0000000000001760
[ 2285.076342] FS: 0000000000000000(0000) GS:ff20f48b3bb80000(0000) knlGS:0000000000000000
[ 2285.077523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2285.078339] CR2: 000054860000343f CR3: 000000011dc10004 CR4: 0000000000771ee0
[ 2285.079316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2285.080295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2285.081272] PKRU: 55555554
[ 2285.081724] Call Trace:
[ 2285.082141] iavf_clean_rx_ring+0xad/0x120 [iavf]
[ 2285.082836] iavf_free_rx_resources+0xe/0x50 [iavf]
[ 2285.083551] iavf_free_all_rx_resources.part.52+0x30/0x50 [iavf]
[ 2285.084393] iavf_reset_task+0x1b3/0x800 [iavf]
[ 2285.085061] process_one_work+0x1a7/0x360
[ 2285.085667] ? create_worker+0x1a0/0x1a0
[ 2285.086258] worker_thread+0x30/0x390
[ 2285.086818] ? create_worker+0x1a0/0x1a0
[ 2285.087401] kthread+0x10b/0x130
[ 2285.087910] ? set_kthread_struct+0x50/0x50
[ 2285.088530] ret_from_fork+0x1f/0x40
Environment
- Red Hat OpenStack Platform 16.2
- Red Hat Enterprise Linux 8.4 EUS/TUS
kernel-4.18.0-305.49.1.el8_4
kernel-4.18.0-305.97.1.el8_4
- SR-IOV VF (Single-Root I/O Virtualization Virtual Function) inside Openstack virtual guest machine
- Intel 700 series NIC using
i40e
andiavf
driver - Large MTU aka Jumbo Frames
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.