After upgrading the kernel, ice/iavf driver and SR-IOV workloads stop working

Solution Verified - Updated -

Issue

  • After upgrading the kernel, the SR-IOV workloads stop functioning and hit the following error in the guest:
iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
  • Network flows are interrupted, the error keeps repeating.
  • Panic occurred with below logs:
[ 2274.923162] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2279.135890] iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 2279.156168] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2285.062016] BUG: unable to handle kernel paging request at 000054860000343f
[ 2285.063051] PGD 0 
[ 2285.063419] Oops: 0002 [#1] SMP NOPTI
[ 2285.063964] CPU: 3 PID: 4861 Comm: kworker/u8:1 Kdump: loaded Tainted: G        W        --------- -  - 4.18.0-425.3.1.el8.x86_64 #1
[ 2285.065586] Hardware name: Red Hat OpenStack Compute, BIOS 1.14.0-1.module+el8.4.0+8855+a9e237a9 04/01/2014
[ 2285.066863] Workqueue: iavf iavf_reset_task [iavf]
[ 2285.067557] RIP: 0010:__page_frag_cache_drain+0x5/0x30
[ 2285.070735] RSP: 0018:ff4c815a0624fdf0 EFLAGS: 00010292
[ 2285.071474] RAX: ffffffff9d5b9740 RBX: ff20f48b074b0000 RCX: 0000000000000002
[ 2285.072441] RDX: 00000000af000000 RSI: 0000000000000000 RDI: 000054860000340b
[ 2285.073412] RBP: ff20f48b0a5d1c00 R08: 0000000000000022 R09: ff20f48b004027e0
[ 2285.074384] R10: 587829f300000000 R11: 0000000000000020 R12: 0000000000000000
[ 2285.075357] R13: 0000000000001000 R14: ff20f48b1dbc0d10 R15: 0000000000001760
[ 2285.076342] FS:  0000000000000000(0000) GS:ff20f48b3bb80000(0000) knlGS:0000000000000000
[ 2285.077523] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2285.078339] CR2: 000054860000343f CR3: 000000011dc10004 CR4: 0000000000771ee0
[ 2285.079316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2285.080295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2285.081272] PKRU: 55555554
[ 2285.081724] Call Trace:
[ 2285.082141]  iavf_clean_rx_ring+0xad/0x120 [iavf]
[ 2285.082836]  iavf_free_rx_resources+0xe/0x50 [iavf]
[ 2285.083551]  iavf_free_all_rx_resources.part.52+0x30/0x50 [iavf]
[ 2285.084393]  iavf_reset_task+0x1b3/0x800 [iavf]
[ 2285.085061]  process_one_work+0x1a7/0x360
[ 2285.085667]  ? create_worker+0x1a0/0x1a0
[ 2285.086258]  worker_thread+0x30/0x390
[ 2285.086818]  ? create_worker+0x1a0/0x1a0
[ 2285.087401]  kthread+0x10b/0x130
[ 2285.087910]  ? set_kthread_struct+0x50/0x50
[ 2285.088530]  ret_from_fork+0x1f/0x40

Environment

  • Red Hat OpenStack Platform 16.2
  • Red Hat Enterprise Linux 8.4 EUS/TUS
  • kernel-4.18.0-305.49.1.el8_4
  • kernel-4.18.0-305.97.1.el8_4
  • SR-IOV VF (Single-Root I/O Virtualization Virtual Function) inside Openstack virtual guest machine
  • Intel 700 series NIC using i40e and iavf driver
  • Large MTU aka Jumbo Frames

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content