Soft lockups on third-party kernel module krg_******_imRH*_K*S_smp64

Issue

• Loading the third-party module krg_xxxxxx_imRH9_K5S_smp64 consistently triggers kernel soft lockup events on the system.
• The server is logging the following messages:

[835472.311287] watchdog: BUG: soft lockup - CPU#5 stuck for 778068s! [ragent:5891]
[835500.312629] watchdog: BUG: soft lockup - CPU#5 stuck for 778094s! [ragent:5891]
[835528.313971] watchdog: BUG: soft lockup - CPU#5 stuck for 778120s! [ragent:5891]
[835556.315313] watchdog: BUG: soft lockup - CPU#5 stuck for 778146s! [ragent:5891]
[835584.316655] watchdog: BUG: soft lockup - CPU#5 stuck for 778172s! [ragent:5891]
[835612.317998] watchdog: BUG: soft lockup - CPU#5 stuck for 778198s! [ragent:5891]
[835640.319346] watchdog: BUG: soft lockup - CPU#5 stuck for 778224s! [ragent:5891]
[835668.320696] watchdog: BUG: soft lockup - CPU#5 stuck for 778250s! [ragent:5891]
[835696.322044] watchdog: BUG: soft lockup - CPU#5 stuck for 778276s! [ragent:5891]
[835724.323392] watchdog: BUG: soft lockup - CPU#5 stuck for 778302s! [ragent:5891]

• The following stack is being recorded:

• For krg_656851_imRH9_K5S_smp64

[835724.323392] watchdog: BUG: soft lockup - CPU#5 stuck for 778302s! [ragent:5891]
[835724.323722] Modules linked in: krg_656851_imRH9_K5S_smp64(PO) bmhook(O) tmhook(O) tls dsa_filter(PO) dsa_filter_hook(O) rfkill vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc binfmt_misc vfat fat intel_rapl_msr intel_rapl_common vmw_balloon intel_uncore_frequency_common nfit libnvdimm vmwgfx rapl pcspkr drm_ttm_helper ttm drm_kms_helper vmw_vmci i2c_piix4 joydev drm xfs libcrc32c sr_mod cdrom sd_mod t10_pi sg crct10dif_pclmul ahci ata_generic crc32_pclmul crc32c_intel libahci ata_piix ghash_clmulni_intel libata vmxnet3 vmw_pvscsi serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
[835724.324719] CPU: 5 PID: 5891 Comm: ragent Kdump: loaded Tainted: P           O L    -------  ---  5.14.0-503.22.1.el9_5.x86_64 #1
[835724.324978] Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.23553139.B64.2403260940 03/26/2024
[835724.325237] RIP: 0010:modify_sct+0x1d/0x30 [krg_656851_imRH9_K5S_smp64]
[835724.325614] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8b 0f 48 89 f0 48 63 d2 25 ff 0f 00 00 48 01 c8 48 85 c9 48 0f 45 f0 <4c> 89 04 d6 c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 90 90 90
[835724.326172] RSP: 0018:ffffacfb475af950 EFLAGS: 00010246
[835724.326436] RAX: 00000000000006a0 RBX: 0000000000000070 RCX: 0000000000000000
[835724.326700] RDX: 000000000000002a RSI: ffffffff842016a0 RDI: ffffffffc13bbc80
[835724.326965] RBP: ffffffffc11d9b70 R08: ffffffffc11d9b70 R09: ffffffff844ebaa0
[835724.327245] R10: 0000000000012f60 R11: 0000000000000000 R12: 000000000000002a
[835724.327509] R13: 000000000000002a R14: 0000000000000150 R15: 0000000000000001
[835724.327768] FS:  00007f43a2475640(0000) GS:ffff95ec8fd40000(0000) knlGS:0000000000000000
[835724.328034] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[835724.328292] CR2: ffffffff842017f0 CR3: 00000001051a4002 CR4: 0000000000770ef0
[835724.328558] PKRU: 55555554
[835724.328821] Call Trace:
[835724.329092]  <IRQ>
[835724.329351]  ? show_trace_log_lvl+0x1c4/0x2df
[835724.329614]  ? show_trace_log_lvl+0x1c4/0x2df
[835724.329877]  ? __pfx_SYS_connect_wrap64+0x10/0x10 [krg_656851_imRH9_K5S_smp64]
[835724.330231]  ? replace_syscall64_internal+0xc0/0x290 [krg_656851_imRH9_K5S_smp64]
[835724.330564]  ? watchdog_timer_fn+0x1ad/0x210
[835724.330830]  ? __pfx_watchdog_timer_fn+0x10/0x10
[835724.331100]  ? __hrtimer_run_queues+0x10f/0x2b0
[835724.331367]  ? hrtimer_interrupt+0xfc/0x210
[835724.331627]  ? __sysvec_apic_timer_interrupt+0x4b/0x100
[835724.331902]  ? sysvec_apic_timer_interrupt+0x6d/0x90
[835724.332173]  </IRQ>
[835724.332435]  <TASK>
[835724.332695]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[835724.332964]  ? __pfx_SYS_connect_wrap64+0x10/0x10 [krg_656851_imRH9_K5S_smp64]
[835724.333306]  ? __pfx_SYS_connect_wrap64+0x10/0x10 [krg_656851_imRH9_K5S_smp64]
[835724.333631]  ? modify_sct+0x1d/0x30 [krg_656851_imRH9_K5S_smp64]
[835724.333958]  replace_syscall64_internal+0xc0/0x290 [krg_656851_imRH9_K5S_smp64]
[835724.334294]  _replace_syscall64_table+0xcf/0x2c0 [krg_656851_imRH9_K5S_smp64]
[835724.334626]  ? check_write_access+0x2f/0x200 [krg_656851_imRH9_K5S_smp64]
[835724.334949]  replace_syscall64+0x43/0x1a0 [krg_656851_imRH9_K5S_smp64]
[835724.335263]  replace_system_calls+0x77/0x1f0 [krg_656851_imRH9_K5S_smp64]
[835724.335574]  driver_hook_system_events+0x1b/0x1b0 [krg_656851_imRH9_K5S_smp64]
[835724.335873]  driver_start_data_collection+0x67/0x3a0 [krg_656851_imRH9_K5S_smp64]
[835724.336172]  krapi_ioctl+0x775/0x2320 [krg_656851_imRH9_K5S_smp64]
[835724.336472]  rntioctl_unlocked+0x19/0x20 [krg_656851_imRH9_K5S_smp64]
[835724.336776]  __x64_sys_ioctl+0x87/0xc0
[835724.337040]  tmhook_invoke+0x45/0x90 [tmhook]
[835724.337307]  tmhook_handler+0x166/0x200 [tmhook]
[835724.337560]  ? do_syscall_64+0x5c/0xf0
[835724.337802]  do_syscall_64+0x5c/0xf0
[835724.338043]  ? syscall_exit_work+0x103/0x130
[835724.338269]  ? syscall_exit_to_user_mode+0x19/0x40
[835724.338488]  ? do_syscall_64+0x6b/0xf0
[835724.338697]  ? syscall_exit_to_user_mode+0x19/0x40
[835724.338906]  ? do_syscall_64+0x6b/0xf0
[835724.339102]  ? __audit_filter_op+0xa5/0xf0
[835724.339293]  ? audit_reset_context.part.0.constprop.0+0x273/0x2e0
[835724.339476]  ? syscall_exit_work+0x103/0x130
[835724.339656]  ? syscall_exit_to_user_mode+0x19/0x40
[835724.339836]  ? do_syscall_64+0x6b/0xf0
[835724.340016]  ? __handle_mm_fault+0x2fb/0x690
[835724.340195]  ? __count_memcg_events+0x4f/0xb0
[835724.340370]  ? mm_account_fault+0x6c/0x100
[835724.340547]  ? handle_mm_fault+0x116/0x270
[835724.340720]  ? do_user_addr_fault+0x1d6/0x6a0
[835724.340897]  ? exc_page_fault+0x62/0x150
[835724.341071]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[835724.341239] RIP: 0033:0x7f43a2f0367b
[835724.341427] Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6d 57 0f 00 f7 d8 64 89 01 48
[835724.341795] RSP: 002b:00007f43a24744d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[835724.341991] RAX: ffffffffffffffda RBX: 00007f439800cd30 RCX: 00007f43a2f0367b
[835724.342185] RDX: 0000000000000000 RSI: 0000000000008001 RDI: 0000000000000011
[835724.342378] RBP: 00007f43a2474520 R08: 00007f4398034ec0 R09: 0000000000000000
[835724.342569] R10: 00007f43a2f9eac0 R11: 0000000000000246 R12: 00000000a2474401
[835724.342761] R13: 00007f43a24744f0 R14: 00007f439800dc7c R15: 0000000000000000
[835724.342954]  </TASK>

• For krg_xxxxxx_imRH8_K2S_smp64

[1928772.163288] watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [ora_dia0_fintrs:2528554]
[1928772.163301] Modules linked in: bmhook(OE) oracleacfs(POE) oracleadvm(POE) oracleoks(POE) overlay krg_656932_imRH8_K2S_smp64(POE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink bridge stp llc vsock_loopback vmw_vsock_virtio_transport_common oracleasm(O) vmw_vsock_vmci_transport vsock sunrpc vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common nfit libnvdimm crc32_pclmul vmw_balloon rapl joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc xfs libcrc32c sr_mod cdrom vmwgfx drm_ttm_helper ttm sd_mod drm_kms_helper t10_pi sg syscopyarea sysfillrect ata_generic ahci crct10dif_pclmul sysimgblt libahci crc32c_intel ata_piix ghash_clmulni_intel drm libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod i2c_dev fuse [last unloaded: bmhook]
[1928772.163462] Red Hat flags: eBPF/event
[1928772.163468] CPU: 3 PID: 2528554 Comm: ora_dia0_fintrs Kdump: loaded Tainted: P           OE     -------- -  - 4.18.0-553.153.1.el8_10.x86_64 #1
[1928772.163476] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[1928772.163480] RIP: 0033:0x4e10dae
[1928772.163511] Code: 00 00 49 89 d5 4d 89 c6 48 8d 55 a0 48 89 4d b0 48 8d 4d a8 48 89 75 98 4c 8d 45 93 49 8b 5d 00 4c 2b cb 49 ff c1 4c 8d 55 80 <ff> 75 38 44 89 4d b8 49 89 fc 49 8b 06 4c 8b 7d 30 4c 8b 4d 28 41
[1928772.163517] RSP: 002b:00007ffffc306620 EFLAGS: 00010216 ORIG_RAX: ffffffffffffff13
[1928772.163524] RAX: 0000000000000001 RBX: 00007ffffc308c38 RCX: 00007ffffc3076a8
[1928772.163528] RDX: 00007ffffc3076a0 RSI: 00007f690cae99c0 RDI: 00007f690caa96d8
[1928772.163532] RBP: 00007ffffc307700 R08: 00007ffffc307693 R09: 0000000000001100
[1928772.163535] R10: 00007ffffc307680 R11: 00007ffffc307778 R12: 00007ffffc30a638
[1928772.163538] R13: 00007ffffc307798 R14: 00007ffffc3077a0 R15: 00007ffffc30a537
[1928772.163541] FS:  00007f690cb3b400 GS:  0000000000000000
[1928772.169304] IPv4: martian source 10.100.25.255 from 10.100.25.231, on dev ens224
[1928772.169310] ll header: 00000000: ff ff ff ff ff ff 00 50 56 a3 e3 d8 08 00        .......PV.....
[1928772.169407] IPv4: martian source 20.20.35.255 from 20.20.35.11, on dev ens192
[1928772.169410] ll header: 00000000: ff ff ff ff ff ff 00 50 56 a3 3a 31 08 00        .......PV.:1..
[1928772.170081] IPv4: martian source 10.100.25.255 from 10.100.25.231, on dev ens224
[1928772.170083] ll header: 00000000: ff ff ff ff ff ff 00 50 56 a3 e3 d8 08 00        .......PV.....
[1928772.170365] IPv4: martian source 20.20.35.255 from 20.20.35.11, on dev ens192
[1928772.170400] ll header: 00000000: ff ff ff ff ff ff 00 50 56 a3 3a 31 08 00        .......PV.:1..
[1928772.170503] sysrq: SysRq : Trigger a crash
[1928772.170507] Kernel panic - not syncing: sysrq triggered crash

[1928772.170509] CPU: 0 PID: 2526045 Comm: cssdmonitor Kdump: loaded Tainted: P           OEL    -------- -  - 4.18.0-553.153.1.el8_10.x86_64 #1
[1928772.170513] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
[1928772.170515] Call Trace:
[1928772.170517]  dump_stack+0x41/0x60
[1928772.170522]  panic+0xe7/0x2ac
[1928772.170526]  ? printk+0x58/0x73
[1928772.170530]  sysrq_handle_crash+0x11/0x20
[1928772.170535]  __handle_sysrq.cold.13+0x48/0xff
[1928772.170538]  write_sysrq_trigger+0x2b/0x40
[1928772.170540]  proc_reg_write+0x39/0x60
[1928772.170544]  vfs_write+0xa5/0x1b0
[1928772.170546]  ksys_write+0x4f/0xb0
[1928772.170548]  tmhook_invoke+0x4a/0xa0 [tmhook]
[1928772.170553]  tmhook_handler+0x1a7/0x210 [tmhook]
[1928772.170556]  ? _generic_fd_read_precode+0x190/0x190 [krg_656932_imRH8_K2S_smp64]
[1928772.170595]  ? _pread_orig+0x2f0/0x2f0 [krg_656932_imRH8_K2S_smp64]
[1928772.170618]  ? writev_post_code+0x5d0/0x5d0 [krg_656932_imRH8_K2S_smp64]
[1928772.170639]  ? _orig_write+0x155/0x220 [krg_656932_imRH8_K2S_smp64]
[1928772.170661]  _orig_write+0x155/0x220 [krg_656932_imRH8_K2S_smp64]
[1928772.170683]  ? _generic_fd_read_precode+0x190/0x190 [krg_656932_imRH8_K2S_smp64]
[1928772.170705]  syscall_wrappers_generic_flow_with_param+0x3e5/0x800 [krg_656932_imRH8_K2S_smp64]
[1928772.170732]  ? syscall_wrappers_generic_write.constprop.69+0x124/0x200 [krg_656932_imRH8_K2S_smp64]
[1928772.170765]  syscall_wrappers_generic_write.constprop.69+0x124/0x200 [krg_656932_imRH8_K2S_smp64]
[1928772.170793]  ? futex_wake+0x78/0x160
[1928772.170796]  SYS_write_common_wrap+0x61/0xd0 [krg_656932_imRH8_K2S_smp64]
[1928772.170821]  ? _pread_orig+0x2f0/0x2f0 [krg_656932_imRH8_K2S_smp64]
[1928772.170845]  do_syscall_64+0x5b/0x1d0
[1928772.170848]  entry_SYSCALL_64_after_hwframe+0x66/0xcb
[1928772.170851] RIP: 0033:0x7f6d690436b7
[1928772.170855] Code: c3 66 90 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 1b fd ff ff 4c 89 e2 48 89 ee 89 df 41 89 c0 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 48 89 44 24 08 e8 54 fd ff ff 48
[1928772.170858] RSP: 002b:00007f6d56c9c650 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[1928772.170860] RAX: ffffffffffffffda RBX: 000000000000001b RCX: 00007f6d690436b7
[1928772.170862] RDX: 0000000000000001 RSI: 00007f6d6ee55ff4 RDI: 000000000000001b
[1928772.170863] RBP: 00007f6d6ee55ff4 R08: 0000000000000000 R09: 00007f6d20000912
[1928772.170864] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000001
[1928772.170865] R13: 00007f6d56c9ca68 R14: 0000000000000002 R15: 000055a47448ac90