After upgrading the kernel, ice/iavf driver and SR-IOV workloads stop working

Solution Verified - Updated -

Issue

  • After upgrading the kernel, the SR-IOV workloads stop functioning and hit the following error in the guest:
iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
  • Network flows are interrupted, the error keeps repeating.
  • Panic occurred with below logs:
[ 2274.923162] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2279.135890] iavf 0000:00:04.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 6
[ 2279.156168] iavf 0000:00:04.0 eth0: NIC Link is Up Speed is 25 Gbps Full Duplex
[ 2285.062016] BUG: unable to handle kernel paging request at 000054860000343f
[ 2285.063051] PGD 0 
[ 2285.063419] Oops: 0002 [#1] SMP NOPTI
[ 2285.063964] CPU: 3 PID: 4861 Comm: kworker/u8:1 Kdump: loaded Tainted: G        W        --------- -  - 4.18.0-425.3.1.el8.x86_64 #1
[ 2285.065586] Hardware name: Red Hat OpenStack Compute, BIOS 1.14.0-1.module+el8.4.0+8855+a9e237a9 04/01/2014
[ 2285.066863] Workqueue: iavf iavf_reset_task [iavf]
[ 2285.067557] RIP: 0010:__page_frag_cache_drain+0x5/0x30
[ 2285.068284] Code: f6 c4 80 74 0f 0f b6 77 51 85 f6 74 07 31 d2 e9 d1 e0 ff ff e9 2c ff ff ff 48 8b 05 05 42 52 01 eb b0 0f 1f 00 0f 1f 44 00 00 <f0> 29 77 34 74 05 e9 60 d6 90 00 48 8b 07 f6 c4 80 74 0f 0f b6 77
[ 2285.070735] RSP: 0018:ff4c815a0624fdf0 EFLAGS: 00010292
[ 2285.071474] RAX: ffffffff9d5b9740 RBX: ff20f48b074b0000 RCX: 0000000000000002
[ 2285.072441] RDX: 00000000af000000 RSI: 0000000000000000 RDI: 000054860000340b
[ 2285.073412] RBP: ff20f48b0a5d1c00 R08: 0000000000000022 R09: ff20f48b004027e0
[ 2285.074384] R10: 587829f300000000 R11: 0000000000000020 R12: 0000000000000000
[ 2285.075357] R13: 0000000000001000 R14: ff20f48b1dbc0d10 R15: 0000000000001760
[ 2285.076342] FS:  0000000000000000(0000) GS:ff20f48b3bb80000(0000) knlGS:0000000000000000
[ 2285.077523] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2285.078339] CR2: 000054860000343f CR3: 000000011dc10004 CR4: 0000000000771ee0
[ 2285.079316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2285.080295] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2285.081272] PKRU: 55555554
[ 2285.081724] Call Trace:
[ 2285.082141]  iavf_clean_rx_ring+0xad/0x120 [iavf]
[ 2285.082836]  iavf_free_rx_resources+0xe/0x50 [iavf]
[ 2285.083551]  iavf_free_all_rx_resources.part.52+0x30/0x50 [iavf]
[ 2285.084393]  iavf_reset_task+0x1b3/0x800 [iavf]
[ 2285.085061]  process_one_work+0x1a7/0x360
[ 2285.085667]  ? create_worker+0x1a0/0x1a0
[ 2285.086258]  worker_thread+0x30/0x390
[ 2285.086818]  ? create_worker+0x1a0/0x1a0
[ 2285.087401]  kthread+0x10b/0x130
[ 2285.087910]  ? set_kthread_struct+0x50/0x50
[ 2285.088530]  ret_from_fork+0x1f/0x40
[ 2285.089079] Modules linked in: nls_utf8 isofs intel_rapl_msr intel_rapl_common nfit libnvdimm cirrus kvm_intel drm_shmem_helper drm_kms_helper kvm syscopyarea sysfillrect irqbypass sysimgblt fb_sys_fops drm pcspkr joydev i2c_piix4 virtio_balloon vfat fat xfs libcrc32c sr_mod cdrom sg ata_generic crct10dif_pclmul crc32_pclmul ata_piix crc32c_intel libata iavf ghash_clmulni_intel serio_raw virtio_blk sunrpc dm_mirror dm_region_hash dm_log dm_mod
[ 2285.094588] CR2: 000054860000343f

Environment

  • Red Hat OpenStack Platform 16.2
  • Red Hat Enterprise Linux 8.4 EUS/TUS
  • kernel-4.18.0-305.49.1.el8_4
  • kernel-4.18.0-305.97.1.el8_4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content