カーネルバージョン 4.18.0-425.3.1.el8.x86_64 を搭載した RHEL8.7 システムが起動に失敗し、ソフトロックアップメッセージが表示される

Issue

• カーネルバージョン 4.18.0-425.3.1.el8.x86_64 を搭載した RHEL8.7 システムは、未署名 (E) ツでリー外 (O) のカーネルモジュールのロード/アンロード中に soft lockup メッセージが表示されて起動に失敗します。

• insmod コマンドを使用して、IBM が出荷した filepath モジュールをロードする際に発生するソフトロックアップ。

  watchdog: BUG: soft lockup - CPU#1 stuck for 134s! [insmod:1141]
  Modules linked in: filepath(OE+) ..
  CPU: 1 PID: 1141 Comm: insmod Tainted: GOE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  RSP: 0000:ffffb4bc43d87c80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffff8dfb78952800 RCX: 0000004e00000000
  RDX: 0000000000000001 RSI: 0000004e00000000 RDI: ffff8dfb20f44a44
  RBP: ffffb4bc43d87cb0 R08: ffffb4bc43d87ce8 R09: ffffffffb9dac900
  R10: ffff8dfac65daa80 R11: 0000000000000001 R12: ffffffffc0885dc0
  R13: ffffffffc0885dd8 R14: ffffffffc0885f90 R15: 0000000000000000
  FS:  00007fda626fe740(0000) GS:ffff8dffe5e40000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007effde4c3ff0 CR3: 0000000108470003 CR4: 00000000001706e0
  Call Trace:
   init_module+0xa7/0x280 [filepath]
   ? driver_ioctl+0x530/0x530 [filepath]
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7fda616139bd
  

• rmmod コマンドを使用して、Commvault が出荷した cvblk モジュールをアンロードする際に発生するソフトロックアップ。

  watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [rmmod:5512]
  Modules linked in: cvblk(POE-) ...
  CPU: 2 PID: 5512 Comm: rmmod Kdump: loaded Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  Code: ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  RSP: 0018:ffffb69f06dcbed8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffffffffc07c6480 RCX: 00000000000072f5
  RDX: 0000000000000001 RSI: f50a3a87f1c29489 RDI: ffffffffc07c88b0
  RBP: ffffb69f06dcbf58 R08: 0000000080000000 R09: ffff9c9168640688
  R10: 0000000000000000 R11: ffffb69f06dcbc40 R12: 0000000000000000
  R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  FS:  00007f28c1649b80(0000) GS:ffff9c9c4de80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000564a7671a098 CR3: 000000015a780000 CR4: 00000000000006e0
  Call Trace:
   cvdrv_utils_destroy+0x2f/0x40 [cvblk]
   __x64_sys_delete_module+0x13d/0x290
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f28c056314b
  

• modprobe コマンドを使用して、Oracle が出荷した oracleafd モジュールをロードする際に発生ソフトロックアップ。

  watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [modprobe:3825]
  Modules linked in: oracleafd(POE+)...
  CPU: 1 PID: 3825 Comm: modprobe Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
  RIP: 0010:native_queued_spin_lock_slowpath+0xf/0x1c0
  Code: 84 c0 75 ce 48 8b 04 24 c6 03 01 48 c7 00 00 00 00 00 e9 69 fe ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 <8b> 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 eb
  RSP: 0000:ffffa54a133e3b80 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000001 RBX: ffffffffc0a5f9e0 RCX: 000000000000001d
  RDX: 0000000000000001 RSI: ffffffffc0a5f040 RDI: ffffffffc0a5fe28
  RBP: ffffa54a133e3c50 R08: 0000000074697865 R09: 0000000000000000
  R10: ffffffffc0a5f140 R11: 0000000000000000 R12: 000000000000003d
  R13: ffffffffc0a5f040 R14: ffffffffc0a5f040 R15: 0000000000000000
  FS:  00007f9704891740(0000) GS:ffff90b517c80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f93fc11235f CR3: 00000003fe6a8002 CR4: 00000000007706e0
  PKRU: 55555554
  Call Trace:
   afdl_writeMsg+0x187/0x320 [oracleafd]
   ? AfdgPrintToBuffer+0x107/0x3f0 [oracleafd]
   AfdgLog+0x129/0x230 [oracleafd]
   afdr_initialize+0xc8/0x210 [oracleafd]
   afd_init+0xde/0x1000 [oracleafd]
   ? 0xffffffffc0a68000
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f97037a69bd
  

• modprobe コマンドを使用して、EMC が出荷した emcpioc モジュールをロードする際に発生するソフトロックアップ。

  watchdog: BUG: soft lockup - CPU#92 stuck for 22s! [modprobe:10805]
  Modules linked in: emcpioc(POE+) ...
  CPU: 92 PID: 10805 Comm: modprobe Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: Dell Inc. PowerEdge R650/0Y2G81, BIOS 1.6.5 04/15/2022
  RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e cd 5b e9 1b b6 aa 00 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
  RSP: 0018:ff398fe26021fb08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000001740001 RBX: ff1d656b11ca6fa0 RCX: 0000000000000000
  RDX: ff1d65777ffabcc0 RSI: 0000000001740000 RDI: ffffffffc1237904
  RBP: 6f4ee0803a438c00 R08: ffffffffc12370e0 R09: ff1d656b11ca6f88
  R10: 0000000000000010 R11: f000000000000000 R12: 2aa9e880b0347b55
  R13: ff398fe26021fb54 R14: 0000000000000020 R15: 0000000000000000
  FS:  00007fdb6cb11740(0000) GS:ff1d65777ff80000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007fdb6baf6470 CR3: 00000035720f8005 CR4: 0000000000771ee0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  PKRU: 55555554
  Call Trace:
   queued_spin_unlock+0x5/0x10 [emcp]
   __raw_spin_unlock+0x5/0x10 [emcp]
   emcp_fetch_unique_dev_id+0xfc/0x110 [emcp]
   emcp_get_scsi_uniq_dev_id+0x442/0x470 [emcp]
   ? apply_workqueue_attrs_locked+0x47/0x80
   emcp_add+0xf7/0x2e0 [emcp]
   ? emcp_add+0x2e0/0x2e0 [emcp]
   emcp_cur_add_device_notify+0x22/0x30 [emcp]
   bus_for_each_dev+0x77/0xc0
   ? down_write+0xe/0x40
   ? 0xffffffffc04c7000
   emcpSetupTraps+0x41/0xb0 [emcp]
   emcp_pioc_daemon+0xda/0x170 [emcp]
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7fdb6ba2c9bd
  

• modprobe コマンドを使用して、Imperva が出荷した krg_621845_imRH8_K2S_smp64 モジュールをロードする際に発生するソフトロックアップ。

  watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [modprobe:15198]
  Modules linked in: krg_621845_imRH8_K2S_smp64(POE+)...
  Red Hat flags: eBPF/rawtrace
  CPU: 20 PID: 15198 Comm: modprobe Kdump: loaded Tainted: POEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  Hardware name: Lenovo ThinkSystem SR650 -[7X06CTO1WW]-/-[7X06CTO1WW]-, BIOS -[IVE152L-2.51]- 01/14/2020
  RIP: 0010:native_queued_spin_lock_slowpath+0x17d/0x1c0
  Code: 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e ed 6f e9 1b b6 aa 00 31 c9 eb 02 f3 90 8b 07 <66> 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01 48 85
  RSP: 0018:ffffb07be67f3bd0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000540001 RBX: ffffffffc1754ab8 RCX: 0000000000000000
  RDX: ffff97d77fa2bcc0 RSI: 0000000000540000 RDI: ffffffffc1754b18
  RBP: ffffffffc1754b38 R08: 00000000000000be R09: 0000000000000001
  R10: 0000000000000000 R11: 0000000000000009 R12: ffffffffc1754b18
  R13: ffffffffc162e92d R14: 0000000000000001 R15: 00000000006000c0
  FS:  00007f36406d2740(0000) GS:ffff97d77fa00000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007f363f6b8470 CR3: 000000410145a004 CR4: 00000000007706e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  PKRU: 55555554
  Call Trace:
   pseudolock_trylock_read+0x4b/0x110 [krg_621845_imRH8_K2S_smp64]
   memory_manager_alloc+0x70/0x480 [krg_621845_imRH8_K2S_smp64]
   init_maintenance_lock+0x86/0x290 [krg_621845_imRH8_K2S_smp64]
   krapi_initialize+0x310a/0x37a0 [krg_621845_imRH8_K2S_smp64]
   ? 0xffffffffc0fb3000
   ra_driver_init+0x14/0x100 [krg_621845_imRH8_K2S_smp64]
   ssphere_init+0x54/0x1000 [krg_621845_imRH8_K2S_smp64]
   ? 0xffffffffc0fb3000
   do_one_initcall+0x46/0x1d0
   ? do_init_module+0x22/0x230
   ? kmem_cache_alloc_trace+0x142/0x280
   do_init_module+0x5a/0x230
   load_module+0x14bf/0x17f0
   ? __do_sys_finit_module+0xb1/0x110
   __do_sys_finit_module+0xb1/0x110
   do_syscall_64+0x5b/0x1b0
   entry_SYSCALL_64_after_hwframe+0x61/0xc6
  RIP: 0033:0x7f363f5ee9bd
  

• insmod コマンドを使用して、Veeam software が出荷した veeamsnap モジュールをロードする際に発生するソフトロックアップ。

  kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [insmod:969322]
  kernel: Modules linked in: veeamsnap(OE+) ... 
  kernel: CPU: 2 PID: 969322 Comm: insmod Tainted: GOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: Xen HVM domU, BIOS 4.13 03/25/2022
  kernel: RIP: 0010:__pv_queued_spin_lock_slowpath+0xe4/0x2a0
  kernel: Code: b6 45 14 41 bd 01 00 00 00 41 be 00 01 00 00 3c 02 0f 94 c0 0f b6 c0 48 89 04 24 c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 0b <f3> 90 83 ea 01 0f 84 5d 01 00 00 0f b6 03 84 c0 75 ee 44 89 f0 f0
  kernel: RSP: 0018:ffffb60984b0fbd0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff0c
  kernel: RAX: 0000000000000003 RBX: ffffffffc06398c0 RCX: 0000000000000000
  kernel: RDX: 0000000000002420 RSI: ffffb60984b0fba0 RDI: 0000000000000003
  kernel: RBP: ffff9f71feaabcc0 R08: ffff9f6f00417908 R09: ffff9f6f00417938
  kernel: R10: 0000000000000000 R11: ffffffff8b7e2c48 R12: 0000000000000000
  kernel: R13: 0000000000000001 R14: 0000000000000100 R15: 00000000000c0000
  kernel: FS:  00007fea9693d740(0000) GS:ffff9f71fea80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 0000564540f0b3a8 CR3: 00000001c37ae006 CR4: 00000000005706e0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: PKRU: 55555554
  kernel: Call Trace:
  kernel: queue_sl_push_back+0x5e/0x70 [veeamsnap]
  kernel: _logging_buffer+0xc5/0x120 [veeamsnap]
  kernel: ? 0xffffffffc05cf000
  kernel: log_s+0x8d/0xb0 [veeamsnap]
  kernel: ? 0xffffffffc05cf000
  kernel: veeamsnap_init+0x36/0x1000 [veeamsnap]
  kernel: do_one_initcall+0x46/0x1d0
  kernel: ? do_init_module+0x22/0x230 
  kernel: ? kmem_cache_alloc_trace+0x142/0x280
  kernel: do_init_module+0x5a/0x230 
  kernel: load_module+0x14bf/0x17f0 
  kernel: ? __do_sys_finit_module+0xb1/0x110
  kernel: __do_sys_finit_module+0xb1/0x110
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  

• Wireguard が出荷した wireguard モジュール用に wg0 デバイスを作成した際に発生するソフトロックアップ。

  # modprobe wireguard
  # ip link add dev wg0 type wireguard
  # dmesg
  kernel: watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [kworker/14:1:186415]
  kernel: Modules linked in: wireguard(E) ...
  kernel: CPU: 14 PID: 186415 Comm: kworker/14:1 Kdump: loaded Tainted: GE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.15.0 06/27/2022
  kernel: Workqueue: events_power_efficient wg_ratelimiter_gc_entries [wireguard]
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  kernel: Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e ad 5d e9 1b b6 aa 00 31 c9 eb 02 f3 90 
  kernel: RSP: 0018:ffffbb6120a37e60 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 00000000003c0001 RBX: ffffffffc0bf7160 RCX: 0000000000000000
  kernel: RDX: ffff9e5adfdabcc0 RSI: 00000000003c0000 RDI: ffffffffc0bf7fb8
  kernel: RBP: 0010708698089b80 R08: ffffffffa49b9ae0 R09: 0000746e65696369
  kernel: R10: 8080808080808080 R11: 0000000000000018 R12: dead000000000200
  kernel: R13: 0000000000000000 R14: ffff9e544e85d500 R15: 0000000000000000
  kernel: FS:  0000000000000000(0000) GS:ffff9e5adfd80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007fc3517c5650 CR3: 00000001cea10001 CR4: 00000000003706e0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: Call Trace:
  kernel:  wg_ratelimiter_gc_entries+0x11b/0x170 [wireguard]
  kernel:  process_one_work+0x1a7/0x360
  kernel:  ? create_worker+0x1a0/0x1a0
  kernel:  worker_thread+0x30/0x390
  kernel:  ? create_worker+0x1a0/0x1a0
  kernel:  kthread+0x10b/0x130
  kernel:  ? set_kthread_struct+0x50/0x50
  kernel:  ret_from_fork+0x35/0x40
  

• rmmod コマンドを使用して、Tripwire が出荷した twnotify モジュールをアンロードする際に発生するソフトロックアップ。

  kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [rmmod:1109860]
  kernel: Modules linked in: twnotify(OE-) 
  kernel: CPU: 0 PID: 1109860 Comm: rmmod Kdump: loaded Tainted: GWOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1 04/01/2014
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffb82ac471fe90 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
  kernel: RDX: 0000000000000001 RSI: ffff99a85fa16698 RDI: ffffffffc0937da8
  kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: c0000000ffff7fff
  kernel: R10: 0000000000000001 R11: ffffb82ac471fcb8 R12: 0000000000000000
  kernel: R13: ffffffffc0937da8 R14: 0000000000000000 R15: 0000000000000000
  kernel: FS:  00007f8a7237a780(0000) GS:ffff99a85fa00000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007f42b88be000 CR3: 0000000291bbe004 CR4: 00000000003706f0
  kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  kernel: Call Trace:
  kernel: __raw_spin_unlock+0x5/0x10 [twnotify]
  kernel: twnotify_set_qlimit.cold.13+0xac/0x131 [twnotify]
  kernel: twnotify_cleanup_module+0x40/0x9e [twnotify]
  kernel: twnotify_exit+0xe1/0x241 [twnotify]
  kernel: __x64_sys_delete_module+0x13d/0x290
  kernel: ? syscall_trace_enter+0x5/0x2d0
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7f8a7108a14b
  kernel: Code: 73 01 c3 48 8b 0d 3d 5d 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d 5d 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007fffadb0c148 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
  kernel: RAX: ffffffffffffffda RBX: 000055ddb4556780 RCX: 00007f8a7108a14b
  kernel: RDX: 0000000000000000 RSI: 0000000000000800 RDI: 000055ddb45567e8
  kernel: RBP: 0000000000000000 R08: 00007fffadb0b0c1 R09: 0000000000000000
  kernel: R10: 00007f8a711c19a0 R11: 0000000000000206 R12: 00007fffadb0c360
  kernel: R13: 00007fffadb0d7a0 R14: 000055ddb45562a0 R15: 000055ddb4556780
  

• insmod コマンドを使用して、Oracle が出荷した oracleoks モジュールをロードする際に発生するソフトロックアップ。

  [  299.461068] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [insmod:1613]   
  [  299.461354] Modules linked in: oracleoks(POE+) ...
  [  299.462882] CPU: 1 PID: 1613 Comm: insmod Kdump: loaded Tainted: POE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  [  299.463273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
  [  299.463464] RIP: 0010:native_queued_spin_lock_slowpath+0xf/0x1c0
  [  299.463667] Code: 84 c0 75 ce 48 8b 04 24 c6 03 01 48 c7 00 00 00 00 00 e9 69 fe ff ff 0f 1f 40 00 66 66 66 66 90 66 66 66 66 90 ba 01 00 00 00 <8b> 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 eb
  [  299.464077] RSP: 0018:ffffa456c0a079a0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  [  299.464326] RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffffffffc0aff508
  [  299.464538] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffc0b4cfe0
  [  299.464739] RBP: ffffffffc0b347a0 R08: 0000000000001b65 R09: 0000000000000000
  [  299.464941] R10: ffffffffc0b348a0 R11: ffffffffc0b347d4 R12: 0000000000000093
  [  299.465160] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000002
  [  299.465392] FS:  00007fac2c3ed740(0000) GS:ffff8d4a1fd00000(0000) knlGS:0000000000000000
  [  299.465602] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  299.465807] CR2: 00007fac2b4550f8 CR3: 000000001ff18003 CR4: 0000000000060ee0
  [  299.466020] Call Trace:
  [  299.466274]  KsReleaseSpinLock_debug+0x19/0x30 [oracleoks]
  [  299.466568]  KsLog+0x633/0x920 [oracleoks]
  [  299.466886]  KS_LOG_TIME_STAMP+0x54/0x70 [oracleoks]
  [  299.467150]  ? printk+0x58/0x73
  [  299.467406]  sKsMsgPrint+0x78/0xc0 [oracleoks]
  [  299.467653]  KsMsgIssue+0x2c5/0x8a0 [oracleoks]
  [  299.467889]  ? KsMalloc+0x8b/0x1e0 [oracleoks]
  [  299.468139]  ? __kmalloc+0x113/0x250
  [  299.468388]  KsLogReinit+0x241/0x2c0 [oracleoks]
  [  299.468625]  oks_init_module+0x15a/0x390 [oracleoks]
  [  299.468862]  ? kcss_rbld_do_step+0x6e0/0x6e0 [oracleoks]
  [  299.469110]  do_one_initcall+0x46/0x1d0
  [  299.469460]  ? do_init_module+0x22/0x230
  [  299.469740]  ? kmem_cache_alloc_trace+0x142/0x280
  [  299.470147]  do_init_module+0x5a/0x230
  [  299.470560]  load_module+0x14bf/0x17f0
  [  299.470792]  ? __do_sys_finit_module+0xb1/0x110
  [  299.471002]  __do_sys_finit_module+0xb1/0x110
  [  299.471254]  do_syscall_64+0x5b/0x1b0
  [  299.471471]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
  [  299.471704] RIP: 0033:0x7fac2b2fe9bd
  

• LINBIT が出荷した drbd モジュールが drbd_thread のセットアップを試みた際に発生するハードロックアップ。

  [  391.688323] Kernel panic - not syncing: Hard LOCKUP
  [  391.688323] CPU: 0 PID: 17089 Comm: drbd_w_testqm Kdump: loaded Tainted: GOE ---------- 4.18.0-425.3.1.el8.x86_64 #1
  [  391.688324] Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.14.2 03/21/2022
  [  391.688324] Call Trace:
  [  391.688324]  <NMI>
  [  391.688324]  dump_stack+0x41/0x60
  [  391.688325]  panic+0xe7/0x2ac
  [  391.688325]  ? __switch_to_asm+0x51/0x80
  [  391.688325]  nmi_panic.cold.11+0xc/0xc
  [  391.688325]  watchdog_overflow_callback.cold.7+0x5c/0x70
  [  391.688326]  __perf_event_overflow+0x52/0x100
  [  391.688326]  handle_pmi_common+0x1f7/0x2c0
  [  391.688326]  ? __set_pte_vaddr+0x32/0x50
  [  391.688327]  ? __native_set_fixmap+0x24/0x40
  [  391.688327]  intel_pmu_handle_irq+0xeb/0x420
  [  391.688327]  perf_event_nmi_handler+0x2d/0x50
  [  391.688327]  nmi_handle+0x63/0x110
  [  391.688328]  default_do_nmi+0x49/0x110
  [  391.688328]  do_nmi+0x1af/0x220
  [  391.688328]  end_repeat_nmi+0x16/0x69
  [  391.688328] RIP: 0010:native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688329] Code: 74 22 48 89 c1 0f 0d 08 eb 20 f3 90 8b 07 85 c0 75 f8 f0 0f b1 17 75 f2 65 ff 0d 6c 5e 8d 65 e9 1b b6 aa 00 31 c9 eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 44 39 c6 74 20 c6 07 01
  [  391.688330] RSP: 0018:ffffb1520ef3be18 EFLAGS: 00000002
  [  391.688330] RAX: 0000000000140001 RBX: ffff9af26b99ab30 RCX: 0000000000000000
  [  391.688331] RDX: ffff9b114002bcc0 RSI: 0000000000040000 RDI: ffff9af26b99ab40
  [  391.688331] RBP: ffffb1520ef3be68 R08: 0000000000000bfa R09: ffff9af26b99ac70
  [  391.688332] R10: 0000000100013bca R11: ffff9b1140029c04 R12: ffff9af26b99ab40
  [  391.688332] R13: ffff9af26b99ab60 R14: ffff9af26b99ac68 R15: ffff9af26b99ac70
  [  391.688333]  ? native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688333]  ? native_queued_spin_lock_slowpath+0x17b/0x1c0
  [  391.688333]  </NMI>
  [  391.688333]  dequeue_work_batch+0x4b/0x70 [drbd]
  [  391.688334]  drbd_worker+0x2fa/0x490 [drbd]
  [  391.688334]  ? __switch_to_asm+0x43/0x80
  [  391.688334]  ? __switch_to+0x10c/0x450
  [  391.688335]  ? __drbd_next_peer_device_ref+0x140/0x140 [drbd]
  [  391.688335]  drbd_thread_setup+0x5e/0x160 [drbd]
  [  391.688335]  ? __drbd_next_peer_device_ref+0x140/0x140 [drbd]
  [  391.688335]  kthread+0x10b/0x130
  [  391.688336]  ? set_kthread_struct+0x50/0x50
  [  391.688336]  ret_from_fork+0x1f/0x40
  

• HPE が出荷した deadman モジュールがロック操作を実行する際に発生するソフトロックアップ。

  kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [cmclconfd:1998]
  kernel: Modules linked in: deadman(OE).. 
  kernel: CPU: 4 PID: 1998 Comm: cmclconfd Tainted: GOE ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffaab98269fe40 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  kernel: RAX: 0000000000000001 RBX: ffffaab98269fe60 RCX: 000000000000000a
  kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffffc06c8850
  kernel: RBP: ffffaab98269fe68 R08: 0000000000000001 R09: 0000000000000000
  kernel: R10: ffffaab98269fed8 R11: ffffffff8dcfa2e0 R12: 0000000000942190
  kernel: R13: 0000000080105730 R14: 0000000000942190 R15: 0000000000000000
  kernel: FS:  00007fa2d1d0f880(0000) GS:ffff9b966fd00000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00000000008f3eb0 CR3: 0000000105636001 CR4: 00000000003706e0
  kernel: Call Trace:
  kernel: current_jiffies_to_tsb.isra.0+0x9d/0xb0 [deadman]
  kernel: deadman_ioctl+0xd9/0x3e0 [deadman]
  kernel: do_vfs_ioctl+0xa4/0x690
  kernel: ksys_ioctl+0x64/0xa0
  kernel: __x64_sys_ioctl+0x16/0x20
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7fa2d05da7cb
  kernel: Code: 73 01 c3 48 8b 0d bd 66 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8d 66 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007ffce52923a8 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
  kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa2d05da7cb
  kernel: RDX: 0000000000942190 RSI: 0000000080105730 RDI: 0000000000000004
  kernel: RBP: 00007ffce52923c0 R08: 0000000000942160 R09: 00007ffce5292388
  kernel: R10: 0000000000000015 R11: 0000000000000202 R12: 0000000000404a30
  kernel: R13: 00007ffce52926b0 R14: 0000000000000000 R15: 0000000000000000
  

• oracleasm モジュールがロック操作を実行する際に発生するソフトロックアップ/rcu ストール。

  kernel: CPU: 2 PID: 1657 Comm: mount Tainted: GOEL ----------- 4.18.0-425.3.1.el8.x86_64 #1
  kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
  kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x24/0x1c0
  kernel: Code: ff ff 0f 1f 40 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 0d f0 0f b1 17 85 c0 75 f2 e9 6e b7 aa 00 f3 90 <eb> e9 81 fe 00 01 00 00 74 44 81 e6 00 ff ff ff 75 71 f0 0f ba 2f
  kernel: RSP: 0018:ffffbd7cd043fd58 EFLAGS: 00000002
  kernel: RAX: 0000000000000001 RBX: ffff99d460b68dc0 RCX: 0000000000000000
  kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff99db1aea9448
  kernel: RBP: ffff99db1aea9440 R08: ffffbd7cd043fd18 R09: ffff99d460b68dc0
  kernel: R10: 00000000006000c0 R11: 0000000000000246 R12: ffff99db1aea9448
  kernel: R13: ffff99db1aea9440 R14: 0000000000000000 R15: ffffffffffffffff
  kernel: FS:  00007fdf14d776c0(0000) GS:ffff99eadfa80000(0000) knlGS:0000000000000000
  kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  kernel: CR2: 00007ffe0dfdcf38 CR3: 000000010b0a0005 CR4: 00000000007706e0
  kernel: PKRU: 55555554
  kernel: Call Trace:
  kernel: asmfs_alloc_inode+0x55/0x90 [oracleasm]
  kernel: ? init_limits.isra.41+0x70/0x70 [oracleasm]
  kernel: alloc_inode+0x1b/0x90
  kernel: new_inode_pseudo+0xc/0x60
  kernel: new_inode+0x12/0x30
  kernel: asmfs_fill_super+0xc3/0x590 [oracleasm]
  kernel: ? sget+0x1cc/0x220
  kernel: ? init_limits.isra.41+0x70/0x70 [oracleasm]
  kernel: mount_nodev+0x49/0xa0
  kernel: legacy_get_tree+0x27/0x50
  kernel: vfs_get_tree+0x25/0xc0
  kernel: do_mount+0x2e9/0x950
  kernel: ? memdup_user+0x4b/0x80
  kernel: ksys_mount+0xbe/0xe0
  kernel: __x64_sys_mount+0x21/0x30
  kernel: do_syscall_64+0x5b/0x1b0
  kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6
  kernel: RIP: 0033:0x7fdf13ce735e
  kernel: Code: 48 8b 0d 2d 5b 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fa 5a 38 00 f7 d8 64 89 01 48
  kernel: RSP: 002b:00007ffe0dfde7a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5
  kernel: RAX: ffffffffffffffda RBX: 0000564e74f88970 RCX: 00007fdf13ce735e
  kernel: RDX: 0000564e74f88b50 RSI: 0000564e74f88b90 RDI: 0000564e74f88b70
  kernel: RBP: 00007fdf14b58184 R08: 0000000000000000 R09: 0000000000000003
  kernel: R10: 00000000c0ed0000 R11: 0000000000000246 R12: 0000000000000000
  kernel: R13: 00000000c0ed0000 R14: 0000564e74f88b70 R15: 0000564e74f88b50
  

• ツリー外 (O) のカーネルモジュールがロック操作を実行する際に発生するソフトロックアップが原因のカーネルパニック。

• ツリー外 (O) のカーネルモジュールがロック操作を実行する際に発生するハードロックアップが原因のカーネルパニック。

• ツリー外 (O) カーネルモジュールがロック操作を実行する際にソフトロックアップメッセージが表示されて発生するシステムハング。

• カーネルスレッドによるハウスキーピングタスクの完了を妨げるソフトロックアップ。