RHEL8.7 system running kernel version 4.18.0-425.10.1.el8_7 hangs or kernel panic due to soft lockup while creating a VDO Volume

Issue

• RHEL8.7 system running kernel version 4.18.0-425.10.1.el8_7 hangs or kernel panic due to soft lockup while creating a VDO Volume.

[  117.227377] uds: loading out-of-tree module taints kernel.
[  117.238530] uds: modprobe: loaded version 8.0.5.1
[  117.266593] kvdo: modprobe: loaded version 6.2.7.17
[  119.708766] kvdo0:dmsetup: underlying device, REQ_FLUSH: supported, REQ_FUA: not supported
[  119.708804] kvdo0:dmsetup: Using write policy async automatically.
[  119.708842] kvdo0:dmsetup: loading device 'vdo1'
[  119.708908] kvdo0:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[  147.477740] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kvdo0:journalQ:1671]
[  147.477776] Modules linked in: kvdo(O) uds(O) dm_bufio nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core virtio_gpu snd_hwdep snd_pcm drm_shmem_helper drm_kms_helper snd_timer snd soundcore syscopyarea sysfillrect sysimgblt fb_sys_fops drm intel_rapl_msr intel_rapl_common i2c_piix4 pcspkr joydev kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl virtio_balloon xfs libcrc32c ata_generic ata_piix libata crc32c_intel serio_raw virtio_scsi virtio_console virtio_blk virtio_net net_failover failover dm_mirror dm_region_hash dm_log dm_mod
[  147.478022] CPU: 1 PID: 1671 Comm: kvdo0:journalQ Kdump: loaded Tainted: GO ----------- 4.18.0-425.10.1.el8_7.x86_64 #1
[  147.478053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
[  147.478073] RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
[  147.478100] Code: ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
[  147.478143] RSP: 0018:ffffa86f40b4fe68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[  147.478162] RAX: 0000000000000001 RBX: 0000000000003ea0 RCX: 0000000000000000
[  147.478179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffffc0b7f038
[  147.478196] RBP: ffff909c847e2100 R08: ffff909cdfd2b978 R09: 000000000000034f
[  147.478213] R10: 000000000000006a R11: 000000000000b9bc R12: ffff909c847e2000
[  147.478230] R13: ffff909c865ea800 R14: ffffffffc0b347f0 R15: ffff909c847e2000
[  147.478247] FS:  0000000000000000(0000) GS:ffff909cdfd00000(0000) knlGS:0000000000000000
[  147.478266] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  147.478281] CR2: 000055f22c8a2a48 CR3: 0000000021610001 CR4: 0000000000060ee0
[  147.478302] Call Trace:
[  147.478321]  _raw_spin_lock+0x1e/0x30
[  147.478346]  initializeWorkQueueStackHandle+0x30/0x70 [kvdo]
[  147.478390]  workQueueRunner+0x4c/0x610 [kvdo]
[  147.478425]  ? dumpSimpleWorkQueue+0x120/0x120 [kvdo]
[  147.478456]  kthread+0x10b/0x130
[  147.478475]  ? set_kthread_struct+0x50/0x50
[  147.478488]  ret_from_fork+0x35/0x40
[  147.478501] Kernel panic - not syncing: softlockup: hung tasks
[  147.478516] CPU: 1 PID: 1671 Comm: kvdo0:journalQ Kdump: loaded Tainted: GOL ----------- 4.18.0-425.10.1.el8_7.x86_64 #1
[  147.478545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
[  147.478865] Call Trace:
[  147.479156]  <IRQ>
[  147.479396]  dump_stack+0x41/0x60
[  147.479648]  panic+0xe7/0x2ac
[  147.479928]  ? __switch_to_asm+0x51/0x80
[  147.480206]  watchdog_timer_fn.cold.10+0x85/0x9e
[  147.480452]  ? watchdog+0x30/0x30
[  147.480678]  __hrtimer_run_queues+0x101/0x280
[  147.480952]  hrtimer_interrupt+0x100/0x220
[  147.481227]  smp_apic_timer_interrupt+0x6a/0x130
[  147.481463]  apic_timer_interrupt+0xf/0x20
[  147.481690]  </IRQ>
[  147.481927] RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
[  147.482310] Code: ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
[  147.482812] RSP: 0018:ffffa86f40b4fe68 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[  147.483082] RAX: 0000000000000001 RBX: 0000000000003ea0 RCX: 0000000000000000
[  147.483314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffffc0b7f038
[  147.483538] RBP: ffff909c847e2100 R08: ffff909cdfd2b978 R09: 000000000000034f
[  147.483810] R10: 000000000000006a R11: 000000000000b9bc R12: ffff909c847e2000
[  147.484088] R13: ffff909c865ea800 R14: ffffffffc0b347f0 R15: ffff909c847e2000
[  147.484316]  ? dumpSimpleWorkQueue+0x120/0x120 [kvdo]
[  147.484564]  ? __switch_to+0x7b/0x450
[  147.484790]  _raw_spin_lock+0x1e/0x30
[  147.485038]  initializeWorkQueueStackHandle+0x30/0x70 [kvdo]
[  147.485295]  workQueueRunner+0x4c/0x610 [kvdo]
[  147.485661]  ? dumpSimpleWorkQueue+0x120/0x120 [kvdo]
[  147.486004]  kthread+0x10b/0x130
[  147.486222]  ? set_kthread_struct+0x50/0x50
[  147.486518]  ret_from_fork+0x35/0x40