System hung with the message 'watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [b9-restart-daem:4083] '.

Issue

• System was not responding and log shows something similar to the below continuously.

...
[Sun Mar 20 07:05:06 EDT 2022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [b9-restart-daem:4083]
[Sun Mar 20 07:05:06 EDT 2022] Modules linked in: nf_tables nfnetlink mptcp_diag xsk_diag vsock_diag tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag b9k_8727(POE) cbproxy_cbp_872_20210
521(POE) cfg80211 rfkill oracleasm(O) vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c intel_rapl_msr intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul g
hash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic sd_mod vmwgfx t10_pi sg drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s
ys_fops ahci ttm libahci drm ata_piix crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi dm_mod fuse
[Sun Mar 20 07:05:06 EDT 2022] CPU: 0 PID: 4083 Comm: b9-restart-daem Kdump: loaded Tainted: P        W  OEL   --------- -  - 4.18.0-305.40.2.el8_4.x86_64 #1
[Sun Mar 20 07:05:06 EDT 2022] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[Sun Mar 20 07:05:06 EDT 2022] RIP: 0010:linuxInterlockedIncrement64+0x18/0x67 [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022] Code: f0 0f c1 01 89 45 ec 8b 55 ec 8b 45 e8 01 d0 90 c9 c3 55 48 89 e5 48 83 ec 30 48 89 7d d0 48 c7 45 d8 01 00 00 00 48 8b 45 d0 <48> 89 45 e0 48 8b 45 e0 be 08 00 00 00 48 89 c
7 e8 43 d6 ff ff 48
[Sun Mar 20 07:05:06 EDT 2022] RSP: 0018:ffffb8368500ba90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[Sun Mar 20 07:05:06 EDT 2022] RAX: ffff96c8bd7ab470 RBX: ffff96c62db74228 RCX: 000000000000003c
[Sun Mar 20 07:05:06 EDT 2022] RDX: ffffffffc0bb03ee RSI: ffff96c62db74280 RDI: ffff96c8bd7ab470
[Sun Mar 20 07:05:06 EDT 2022] RBP: ffffb8368500bac0 R08: 0015c4dfbbe385a2 R09: ffff96c8b535f718
[Sun Mar 20 07:05:06 EDT 2022] R10: 732f6e69622f7273 R11: 6c74636d65747379 R12: ffffb83683c5bba0
[Sun Mar 20 07:05:06 EDT 2022] R13: ffff96c97f64af80 R14: ffffffffc0ba57dd R15: 0000000000000000
[Sun Mar 20 07:05:06 EDT 2022] FS:  0000000000000000(0000) GS:ffff96ca3fa00000(0000) knlGS:0000000000000000
[Sun Mar 20 07:05:06 EDT 2022] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Mar 20 07:05:06 EDT 2022] CR2: 000055d691b61fe8 CR3: 0000000827e10006 CR4: 00000000001706f0
[Sun Mar 20 07:05:06 EDT 2022] Call Trace:
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN11PeakCounteraSEy+0x2f/0x3a [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  _ZN7CounterppEv+0x1c/0x1e [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN9HashTableI13WatchdogEntryE12LockedInsertEPS0_P10TableEntryIS0_E+0xec/0x10c [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN12GenericTableI13WatchdogEntryE6InsertEPS0_P10TableEntryIS0_E+0x36/0x38 [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN16WatchdogServices11CreateEntryERbPKc+0xae/0x264 [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN8WatchdogC2EPKcNS_16WatchdogFeaturesEP16WatchdogServices+0xf2/0x158 [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? _ZN15ProcessTracking28TerminateProcessNotificationEiP11ProcessInfo+0x36/0x25c [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? prefilt_exit+0x24/0x2b [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? run_parity_process+0xe93/0xed4 [b9k_8727]
[Sun Mar 20 07:05:06 EDT 2022]  ? finish_wait+0x80/0x80
[Sun Mar 20 07:05:06 EDT 2022]  ? kthread+0x116/0x130
[Sun Mar 20 07:05:06 EDT 2022]  ? kthread_flush_work_fn+0x10/0x10
[Sun Mar 20 07:05:06 EDT 2022]  ? ret_from_fork+0x35/0x40
...