RHEL8.7 system with the kernel version 4.18.0-425.3.1.el8.x86_64 fails to boot with soft lockup message

Issue

• RHEL8.7 system with the kernel version 4.18.0-425.3.1.el8.x86_64 fails to boot with soft lockup messages during loading/unloading of an unsigned (E) Out-of-tree (O) kernel module.

• Soft lockup while loading the IBM shipped [filepath] module using the insmod command.

  watchdog: BUG: soft lockup - CPU#1 stuck for 134s! [insmod:1141]
  Modules linked in: filepath(OE+) ..
  CPU: 1 PID: 1141 Comm: insmod Tainted: GOE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  RSP: 0000:ffffb4bc43d87c80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffff8dfb78952800 RCX: 0000004e00000000
  RDX: 0000000000000001 RSI: 0000004e00000000 RDI: ffff8dfb20f44a44
  RBP: ffffb4bc43d87cb0 R08: ffffb4bc43d87ce8 R09: ffffffffb9dac900
  R10: ffff8dfac65daa80 R11: 0000000000000001 R12: ffffffffc0885dc0
  R13: ffffffffc0885dd8 R14: ffffffffc0885f90 R15: 0000000000000000
  FS:  00007fda626fe740(0000) GS:ffff8dffe5e40000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007effde4c3ff0 CR3: 0000000108470003 CR4: 00000000001706e0
  Call Trace:
   init_module+0xa7/0x280 [filepath]
   ? driver_ioctl+0x530/0x530 [filepath]
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7fda616139bd
  

• Soft lockup while unloading the Commvault shipped [cvblk] module using the rmmod command.

  watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [rmmod:5512]
  Modules linked in: cvblk(POE-) ...
  CPU: 2 PID: 5512 Comm: rmmod Kdump: loaded Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  Code: ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  RSP: 0018:ffffb69f06dcbed8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffffffffc07c6480 RCX: 00000000000072f5
  RDX: 0000000000000001 RSI: f50a3a87f1c29489 RDI: ffffffffc07c88b0
  RBP: ffffb69f06dcbf58 R08: 0000000080000000 R09: ffff9c9168640688
  R10: 0000000000000000 R11: ffffb69f06dcbc40 R12: 0000000000000000
  R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  FS:  00007f28c1649b80(0000) GS:ffff9c9c4de80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000564a7671a098 CR3: 000000015a780000 CR4: 00000000000006e0
  Call Trace:
   cvdrv_utils_destroy+0x2f/0x40 [cvblk]
   __x64_sys_delete_module+0x13d/0x290
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f28c056314b
  

• Soft lockup while loading the Oracle shipped [oracleafd] module using the modprobe command.

  watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [modprobe:3825]
  Modules linked in: oracleafd(POE+)...
  CPU: 1 PID: 3825 Comm: modprobe Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
  RIP: 0010:native_queued_spin_lock_slowpath+0xf/0x1c0
  Code: 84 c0 75 ce 48 8b 04 24 c6 03 01 48 c7 00 00 00 00 00 e9 69 fe ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 <8b> 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 eb
  RSP: 0000:ffffa54a133e3b80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffffffffc0a5f9e0 RCX: 000000000000001d
  RDX: 0000000000000001 RSI: ffffffffc0a5f040 RDI: ffffffffc0a5fe28
  RBP: ffffa54a133e3c50 R08: 0000000074697865 R09: 0000000000000000
  R10: ffffffffc0a5f140 R11: 0000000000000000 R12: 000000000000003d
  R13: ffffffffc0a5f040 R14: ffffffffc0a5f040 R15: 0000000000000000
  FS:  00007f9704891740(0000) GS:ffff90b517c80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f93fc11235f CR3: 00000003fe6a8002 CR4: 00000000007706e0
  PKRU: 55555554
  Call Trace:
   afdl_writeMsg+0x187/0x320 [oracleafd]
   ? AfdgPrintToBuffer+0x107/0x3f0 [oracleafd]
   AfdgLog+0x129/0x230 [oracleafd]
   afdr_initialize+0xc8/0x210 [oracleafd]
   afd_init+0xde/0x1000 [oracleafd]
   ? 0xffffffffc0a68000
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f97037a69bd
  

• Soft lockup while loading the EMC shipped [emcpioc] module using the modprobe command.

  watchdog: BUG: soft lockup - CPU#92 stuck for 22s! [modprobe:10805]
  Modules linked in: emcpioc(POE+) ...
  CPU: 92 PID: 10805 Comm: modprobe Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: Dell Inc. PowerEdge R650/0Y2G81, BIOS 1.6.5 04/15/2022
  RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e cd 5b e9 1b b6 aa 00 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
  RSP: 0018:ff398fe26021fb08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000001740001 RBX: ff1d656b11ca6fa0 RCX: 0000000000000000
  RDX: ff1d65777ffabcc0 RSI: 0000000001740000 RDI: ffffffffc1237904
  RBP: 6f4ee0803a438c00 R08: ffffffffc12370e0 R09: ff1d656b11ca6f88
  R10: 0000000000000010 R11: f000000000000000 R12: 2aa9e880b0347b55
  R13: ff398fe26021fb54 R14: 0000000000000020 R15: 0000000000000000
  FS:  00007fdb6cb11740(0000) GS:ff1d65777ff80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007fdb6baf6470 CR3: 00000035720f8005 CR4: 0000000000771ee0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  PKRU: 55555554
  Call Trace:
   queued_spin_unlock+0x5/0x10 [emcp]
   __raw_spin_unlock+0x5/0x10 [emcp]
   emcp_fetch_unique_dev_id+0xfc/0x110 [emcp]
   emcp_get_scsi_uniq_dev_id+0x442/0x470 [emcp]
   ? apply_workqueue_attrs_locked+0x47/0x80
   emcp_add+0xf7/0x2e0 [emcp]
   ? emcp_add+0x2e0/0x2e0 [emcp]
   emcp_cur_add_device_notify+0x22/0x30 [emcp]
   bus_for_each_dev+0x77/0xc0
   ? down_write+0xe/0x40
   ? 0xffffffffc04c7000
   emcpSetupTraps+0x41/0xb0 [emcp]
   emcp_pioc_daemon+0xda/0x170 [emcp]
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7fdb6ba2c9bd
  

• Soft lockup while loading the Imperva shipped [krg_621845_imRH8_K2S_smp64] module using the modprobe command.

  watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [modprobe:15198]
  Modules linked in: krg_621845_imRH8_K2S_smp64(POE+)...
  Red Hat flags: eBPF/rawtrace
  CPU: 20 PID: 15198 Comm: modprobe Kdump: loaded Tainted: POEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: Lenovo ThinkSystem SR650 -[7X06CTO1WW]-/-[7X06CTO1WW]-, BIOS -[IVE152L-2.51]- 01/14/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x17d/0x1c0
  Code: 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e ed 6f e9 1b b6 aa 00 31 c9 eb 02 f3 90 8b 07 <66> 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01 48 85
  RSP: 0018:ffffb07be67f3bd0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000540001 RBX: ffffffffc1754ab8 RCX: 0000000000000000
  RDX: ffff97d77fa2bcc0 RSI: 0000000000540000 RDI: ffffffffc1754b18
  RBP: ffffffffc1754b38 R08: 00000000000000be R09: 0000000000000001
  R10: 0000000000000000 R11: 0000000000000009 R12: ffffffffc1754b18
  R13: ffffffffc162e92d R14: 0000000000000001 R15: 00000000006000c0
  FS:  00007f36406d2740(0000) GS:ffff97d77fa00000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f363f6b8470 CR3: 000000410145a004 CR4: 00000000007706e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  PKRU: 55555554
  Call Trace:
   pseudolock_trylock_read+0x4b/0x110 [krg_621845_imRH8_K2S_smp64]
   memory_manager_alloc+0x70/0x480 [krg_621845_imRH8_K2S_smp64]
   init_maintenance_lock+0x86/0x290 [krg_621845_imRH8_K2S_smp64]
   krapi_initialize+0x310a/0x37a0 [krg_621845_imRH8_K2S_smp64]
   ? 0xffffffffc0fb3000
   ra_driver_init+0x14/0x100 [krg_621845_imRH8_K2S_smp64]
   ssphere_init+0x54/0x1000 [krg_621845_imRH8_K2S_smp64]
   ? 0xffffffffc0fb3000
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f363f5ee9bd
  

• Soft lockup while loading the Veeam software shipped [veeamsnap] module using the insmod command.

  kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [insmod:969322]
  kernel: Modules linked in: veeamsnap(OE+) ... 
  kernel: CPU: 2 PID: 969322 Comm: insmod Tainted: GOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: Xen HVM domU, BIOS 4.13 03/25/2022
  kernel: RIP: 0010:__pv_queued_spin_lock_slowpath+0xe4/0x2a0
  kernel: Code: b6 45 14 41 bd 01 00 00 00 41 be 00 01 00 00 3c 02 0f 94 c0 0f b6 c0 48 89 04 24 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 0b <f3> 90 83 ea 01 0f 84 5d 01 00 00 0f b6 03 84 c0 75 ee 44 89 f0 f0
  kernel: RSP: 0018:ffffb60984b0fbd0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff0c
  kernel: RAX: 0000000000000003 RBX: ffffffffc06398c0 RCX: 0000000000000000
  kernel: RDX: 0000000000002420 RSI: ffffb60984b0fba0 RDI: 0000000000000003
  kernel: RBP: ffff9f71feaabcc0 R08: ffff9f6f00417908 R09: ffff9f6f00417938
  kernel: R10: 0000000000000000 R11: ffffffff8b7e2c48 R12: 0000000000000000
  kernel: R13: 0000000000000001 R14: 0000000000000100 R15: 00000000000c0000
  kernel: FS:  00007fea9693d740(0000) GS:ffff9f71fea80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 0000564540f0b3a8 CR3: 00000001c37ae006 CR4: 00000000005706e0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: PKRU: 55555554
  kernel: Call Trace:
  kernel: queue_sl_push_back+0x5e/0x70 [veeamsnap]
  kernel: _logging_buffer+0xc5/0x120 [veeamsnap]
  kernel: ? 0xffffffffc05cf000
  kernel: log_s+0x8d/0xb0 [veeamsnap]
  kernel: ? 0xffffffffc05cf000
  kernel: veeamsnap_init+0x36/0x1000 [veeamsnap]
  kernel: do_one_initcall+0x46/0x1d0
  kernel: ? do_init_module+0x22/0x230 
  kernel: ? kmem_cache_alloc_trace+0x142/0x280
  kernel: do_init_module+0x5a/0x230 
  kernel: load_module+0x14bf/0x17f0 
  kernel: ? __do_sys_finit_module+0xb1/0x110
  kernel: __do_sys_finit_module+0xb1/0x110
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  

• Soft lockup when wg0 device is created for Wireguard shipped [wireguard] module.

  # modprobe wireguard
  # ip link add dev wg0 type wireguard
  # dmesg
  kernel: watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [kworker/14:1:186415]
  kernel: Modules linked in: wireguard(E) ...
  kernel: CPU: 14 PID: 186415 Comm: kworker/14:1 Kdump: loaded Tainted: GE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.15.0 06/27/2022
  kernel: Workqueue: events_power_efficient wg_ratelimiter_gc_entries [wireguard]
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  kernel: Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e ad 5d e9 1b b6 aa 00 31 c9 eb 02 f3 90 
  kernel: RSP: 0018:ffffbb6120a37e60 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 00000000003c0001 RBX: ffffffffc0bf7160 RCX: 0000000000000000
  kernel: RDX: ffff9e5adfdabcc0 RSI: 00000000003c0000 RDI: ffffffffc0bf7fb8
  kernel: RBP: 0010708698089b80 R08: ffffffffa49b9ae0 R09: 0000746e65696369
  kernel: R10: 8080808080808080 R11: 0000000000000018 R12: dead000000000200
  kernel: R13: 0000000000000000 R14: ffff9e544e85d500 R15: 0000000000000000
  kernel: FS:  0000000000000000(0000) GS:ffff9e5adfd80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007fc3517c5650 CR3: 00000001cea10001 CR4: 00000000003706e0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: Call Trace:
  kernel:  wg_ratelimiter_gc_entries+0x11b/0x170 [wireguard]
  kernel:  process_one_work+0x1a7/0x360
  kernel:  ? create_worker+0x1a0/0x1a0
  kernel:  worker_thread+0x30/0x390
  kernel:  ? create_worker+0x1a0/0x1a0
  kernel:  kthread+0x10b/0x130
  kernel:  ? set_kthread_struct+0x50/0x50
  kernel:  ret_from_fork+0x35/0x40
  

• Soft lockup while unloading the Tripwire shipped [twnotify] module using the rmmod command.

  kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [rmmod:1109860]
  kernel: Modules linked in: twnotify(OE-) 
  kernel: CPU: 0 PID: 1109860 Comm: rmmod Kdump: loaded Tainted: GWOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1 04/01/2014
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffb82ac471fe90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
  kernel: RDX: 0000000000000001 RSI: ffff99a85fa16698 RDI: ffffffffc0937da8
  kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: c0000000ffff7fff
  kernel: R10: 0000000000000001 R11: ffffb82ac471fcb8 R12: 0000000000000000
  kernel: R13: ffffffffc0937da8 R14: 0000000000000000 R15: 0000000000000000
  kernel: FS:  00007f8a7237a780(0000) GS:ffff99a85fa00000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007f42b88be000 CR3: 0000000291bbe004 CR4: 00000000003706f0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: Call Trace:
  kernel: __raw_spin_unlock+0x5/0x10 [twnotify]
  kernel: twnotify_set_qlimit.cold.13+0xac/0x131 [twnotify]
  kernel: twnotify_cleanup_module+0x40/0x9e [twnotify]
  kernel: twnotify_exit+0xe1/0x241 [twnotify]
  kernel: __x64_sys_delete_module+0x13d/0x290
  kernel: ? syscall_trace_enter+0x5/0x2d0
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7f8a7108a14b
  kernel: Code: 73 01 c3 48 8b 0d 3d 5d 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d 5d 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007fffadb0c148 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
  kernel: RAX: ffffffffffffffda RBX: 000055ddb4556780 RCX: 00007f8a7108a14b
  kernel: RDX: 0000000000000000 RSI: 0000000000000800 RDI: 000055ddb45567e8
  kernel: RBP: 0000000000000000 R08: 00007fffadb0b0c1 R09: 0000000000000000
  kernel: R10: 00007f8a711c19a0 R11: 0000000000000206 R12: 00007fffadb0c360
  kernel: R13: 00007fffadb0d7a0 R14: 000055ddb45562a0 R15: 000055ddb4556780
  

• Soft lockup while loading the Oracle shipped [oracleoks] module using the insmod command.

  [  299.461068] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [insmod:1613]   
  [  299.461354] Modules linked in: oracleoks(POE+) ...
  [  299.462882] CPU: 1 PID: 1613 Comm: insmod Kdump: loaded Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  [  299.463273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
  [  299.463464] RIP: 0010:native_queued_spin_lock_slowpath+0xf/0x1c0
  [  299.463667] Code: 84 c0 75 ce 48 8b 04 24 c6 03 01 48 c7 00 00 00 00 00 e9 69 fe ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 eb
  [  299.464077] RSP: 0018:ffffa456c0a079a0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  [  299.464326] RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffffffffc0aff508
  [  299.464538] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffc0b4cfe0
  [  299.464739] RBP: ffffffffc0b347a0 R08: 0000000000001b65 R09: 0000000000000000
  [  299.464941] R10: ffffffffc0b348a0 R11: ffffffffc0b347d4 R12: 0000000000000093
  [  299.465160] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000002
  [  299.465392] FS:  00007fac2c3ed740(0000) GS:ffff8d4a1fd00000(0000) knlGS:0000000000000000
  [  299.465602] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  299.465807] CR2: 00007fac2b4550f8 CR3: 000000001ff18003 CR4: 0000000000060ee0
  [  299.466020] Call Trace:
  [  299.466274]  KsReleaseSpinLock_debug+0x19/0x30 [oracleoks]
  [  299.466568]  KsLog+0x633/0x920 [oracleoks]
  [  299.466886]  KS_LOG_TIME_STAMP+0x54/0x70 [oracleoks]
  [  299.467150]  ? printk+0x58/0x73
  [  299.467406]  sKsMsgPrint+0x78/0xc0 [oracleoks]
  [  299.467653]  KsMsgIssue+0x2c5/0x8a0 [oracleoks]
  [  299.467889]  ? KsMalloc+0x8b/0x1e0 [oracleoks]
  [  299.468139]  ? __kmalloc+0x113/0x250
  [  299.468388]  KsLogReinit+0x241/0x2c0 [oracleoks]
  [  299.468625]  oks_init_module+0x15a/0x390 [oracleoks]
  [  299.468862]  ? kcss_rbld_do_step+0x6e0/0x6e0 [oracleoks]
  [  299.469110]  do_one_initcall+0x46/0x1d0
  [  299.469460]  ? do_init_module+0x22/0x230
  [  299.469740]  ? kmem_cache_alloc_trace+0x142/0x280
  [  299.470147]  do_init_module+0x5a/0x230
  [  299.470560]  load_module+0x14bf/0x17f0
  [  299.470792]  ? __do_sys_finit_module+0xb1/0x110
  [  299.471002]  __do_sys_finit_module+0xb1/0x110
  [  299.471254]  do_syscall_64+0x5b/0x1b0
  [  299.471471]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
  [  299.471704] RIP: 0033:0x7fac2b2fe9bd
  

• Hard lockup while the LINBIT shipped [drbd] module was trying to setup drbd_thread.

  [  391.688323] Kernel panic - not syncing: Hard LOCKUP
  [  391.688323] CPU: 0 PID: 17089 Comm: drbd_w_testqm Kdump: loaded Tainted: GOE ---------- 4.18.0-425.3.1.el8.x86_64 #1
  [  391.688324] Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.14.2 03/21/2022
  [  391.688324] Call Trace:
  [  391.688324]  <NMI>
  [  391.688324]  dump_stack+0x41/0x60
  [  391.688325]  panic+0xe7/0x2ac
  [  391.688325]  ? __switch_to_asm+0x51/0x80
  [  391.688325]  nmi_panic.cold.11+0xc/0xc
  [  391.688325]  watchdog_overflow_callback.cold.7+0x5c/0x70
  [  391.688326]  __perf_event_overflow+0x52/0x100
  [  391.688326]  handle_pmi_common+0x1f7/0x2c0
  [  391.688326]  ? __set_pte_vaddr+0x32/0x50
  [  391.688327]  ? __native_set_fixmap+0x24/0x40
  [  391.688327]  intel_pmu_handle_irq+0xeb/0x420
  [  391.688327]  perf_event_nmi_handler+0x2d/0x50
  [  391.688327]  nmi_handle+0x63/0x110
  [  391.688328]  default_do_nmi+0x49/0x110
  [  391.688328]  do_nmi+0x1af/0x220
  [  391.688328]  end_repeat_nmi+0x16/0x69
  [  391.688328] RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688329] Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e 8d 65 e9 1b b6 aa 00 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
  [  391.688330] RSP: 0018:ffffb1520ef3be18 EFLAGS: 00000002
  [  391.688330] RAX: 0000000000140001 RBX: ffff9af26b99ab30 RCX: 0000000000000000
  [  391.688331] RDX: ffff9b114002bcc0 RSI: 0000000000040000 RDI: ffff9af26b99ab40
  [  391.688331] RBP: ffffb1520ef3be68 R08: 0000000000000bfa R09: ffff9af26b99ac70
  [  391.688332] R10: 0000000100013bca R11: ffff9b1140029c04 R12: ffff9af26b99ab40
  [  391.688332] R13: ffff9af26b99ab60 R14: ffff9af26b99ac68 R15: ffff9af26b99ac70
  [  391.688333]  ? native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688333]  ? native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688333]  </NMI>
  [  391.688333]  dequeue_work_batch+0x4b/0x70 [drbd]
  [  391.688334]  drbd_worker+0x2fa/0x490 [drbd]
  [  391.688334]  ? __switch_to_asm+0x43/0x80
  [  391.688334]  ? __switch_to+0x10c/0x450
  [  391.688335]  ? __drbd_next_peer_device_ref+0x140/0x140 [drbd]
  [  391.688335]  drbd_thread_setup+0x5e/0x160 [drbd]
  [  391.688335]  ? __drbd_next_peer_device_ref+0x140/0x140 [drbd]
  [  391.688335]  kthread+0x10b/0x130
  [  391.688336]  ? set_kthread_struct+0x50/0x50
  [  391.688336]  ret_from_fork+0x1f/0x40
  

• Soft lockup while the HPE shipped [deadman] module was performing a locking operation.

  kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [cmclconfd:1998]
  kernel: Modules linked in: deadman(OE).. 
  kernel: CPU: 4 PID: 1998 Comm: cmclconfd Tainted: GOE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffaab98269fe40 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 0000000000000001 RBX: ffffaab98269fe60 RCX: 000000000000000a
  kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffffc06c8850
  kernel: RBP: ffffaab98269fe68 R08: 0000000000000001 R09: 0000000000000000
  kernel: R10: ffffaab98269fed8 R11: ffffffff8dcfa2e0 R12: 0000000000942190
  kernel: R13: 0000000080105730 R14: 0000000000942190 R15: 0000000000000000
  kernel: FS:  00007fa2d1d0f880(0000) GS:ffff9b966fd00000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00000000008f3eb0 CR3: 0000000105636001 CR4: 00000000003706e0
  kernel: Call Trace:
  kernel: current_jiffies_to_tsb.isra.0+0x9d/0xb0 [deadman]
  kernel: deadman_ioctl+0xd9/0x3e0 [deadman]
  kernel: do_vfs_ioctl+0xa4/0x690
  kernel: ksys_ioctl+0x64/0xa0
  kernel: __x64_sys_ioctl+0x16/0x20
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7fa2d05da7cb
  kernel: Code: 73 01 c3 48 8b 0d bd 66 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8d 66 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007ffce52923a8 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
  kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa2d05da7cb
  kernel: RDX: 0000000000942190 RSI: 0000000080105730 RDI: 0000000000000004
  kernel: RBP: 00007ffce52923c0 R08: 0000000000942160 R09: 00007ffce5292388
  kernel: R10: 0000000000000015 R11: 0000000000000202 R12: 0000000000404a30
  kernel: R13: 00007ffce52926b0 R14: 0000000000000000 R15: 0000000000000000
  

• Soft lockup/rcu stall while the [oracleasm] module was performing a locking operation.

  kernel: CPU: 2 PID: 1657 Comm: mount Tainted: GOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffbd7cd043fd58 EFLAGS: 00000002
  kernel: RAX: 0000000000000001 RBX: ffff99d460b68dc0 RCX: 0000000000000000
  kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff99db1aea9448
  kernel: RBP: ffff99db1aea9440 R08: ffffbd7cd043fd18 R09: ffff99d460b68dc0
  kernel: R10: 00000000006000c0 R11: 0000000000000246 R12: ffff99db1aea9448
  kernel: R13: ffff99db1aea9440 R14: 0000000000000000 R15: ffffffffffffffff
  kernel: FS:  00007fdf14d776c0(0000) GS:ffff99eadfa80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007ffe0dfdcf38 CR3: 000000010b0a0005 CR4: 00000000007706e0
  kernel: PKRU: 55555554
  kernel: Call Trace:
  kernel: asmfs_alloc_inode+0x55/0x90 [oracleasm]
  kernel: ? init_limits.isra.41+0x70/0x70 [oracleasm]
  kernel: alloc_inode+0x1b/0x90
  kernel: new_inode_pseudo+0xc/0x60
  kernel: new_inode+0x12/0x30
  kernel: asmfs_fill_super+0xc3/0x590 [oracleasm]
  kernel: ? sget+0x1cc/0x220
  kernel: ? init_limits.isra.41+0x70/0x70 [oracleasm]
  kernel: mount_nodev+0x49/0xa0
  kernel: legacy_get_tree+0x27/0x50
  kernel: vfs_get_tree+0x25/0xc0
  kernel: do_mount+0x2e9/0x950
  kernel: ? memdup_user+0x4b/0x80
  kernel: ksys_mount+0xbe/0xe0
  kernel: __x64_sys_mount+0x21/0x30
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7fdf13ce735e
  kernel: Code: 48 8b 0d 2d 5b 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fa 5a 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007ffe0dfde7a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
  kernel: RAX: ffffffffffffffda RBX: 0000564e74f88970 RCX: 00007fdf13ce735e
  kernel: RDX: 0000564e74f88b50 RSI: 0000564e74f88b90 RDI: 0000564e74f88b70
  kernel: RBP: 00007fdf14b58184 R08: 0000000000000000 R09: 0000000000000003
  kernel: R10: 00000000c0ed0000 R11: 0000000000000246 R12: 0000000000000000
  kernel: R13: 00000000c0ed0000 R14: 0000564e74f88b70 R15: 0000564e74f88b50
  

• Kernel panic due to soft lockup while an out-of-tree (O) kernel module is performing a locking operation.

• Kernel panic due to hard lockup while an out-of-tree (O) kernel module is performing a locking operation.

• System hangs with a soft lockup message while an out-of-tree (O) kernel module performs a locking operation.

• Soft lockup preventing kernel threads from completing their housekeeping tasks.