Kernel panic in the mutex_lock()/__mutex_lock() due to a third-party kernel module [falcon_lsm_serviceable]

Issue

• RHEL8 system crashed in the __mutex_lock() function with the following call traces.

[77384.462408] BUG: unable to handle kernel paging request at 0000073321736b48
[77384.462794] PGD 0 P4D 0 
[77384.463122] Oops: 0000 [#1] SMP PTI
[77384.463372] CPU: 3 PID: 81443 Comm: systemd-journal Tainted: PE ----------- 4.18.0-513.18.1.el8_9.x86_64 #1
[77384.463641] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
[77384.463923] RIP: 0010:__mutex_lock.isra.11+0xbe/0x420
[77384.464171] Code: c3 cc cc cc cc 48 89 fe 48 89 c2 eb bb 65 48 8b 04 25 40 dc 01 00 48 8b 00 a8 08 75 22 48 8b 03 48 83 e0 f8 0f 84 90 00 00 00 <8b> 50 38 85 d2 74 0e 8b 78 3c 31 c0 0f 1f 44 00 00 84 c0 74 7b 65
[77384.464672] RSP: 0000:ffffb7f0c4da7dd0 EFLAGS: 00010202
[77384.464912] RAX: 0000073321736b10 RBX: ffff8a2ae8025d38 RCX: 0000073321736b10
[77384.465157] RDX: ffff8a2739ba8000 RSI: 0000000000000002 RDI: ffff8a2739ba8000
[77384.465391] RBP: ffffb7f0c4da7e30 R08: 0000000000800000 R09: 0000000000800000
[77384.465632] R10: ffffffffc0ac7a20 R11: 0000000000000800 R12: 0000000000000001
[77384.465861] R13: ffffb7f0c4da7ed0 R14: 0000000000000002 R15: ffff8a2ae8025cf8
[77384.466086] FS:  00007fa27b7b2980(0000) GS:ffff8a2775ec0000(0000) knlGS:0000000000000000
[77384.466310] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[77384.466531] CR2: 0000073321736b48 CR3: 000000010e4a6001 CR4: 00000000003706e0
[77384.466773] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[77384.466991] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[77384.467200] Call Trace:
[77384.467398]  ? __die_body+0x1a/0x60
[77384.467601]  ? no_context+0x1ba/0x3f0
[77384.467788]  ? __bad_area_nosemaphore+0x16c/0x1c0
[77384.468023]  ? do_page_fault+0x37/0x12d
[77384.468203]  ? page_fault+0x1e/0x30
[77384.468388]  ? __mutex_lock.isra.11+0xbe/0x420
[77384.468576]  crowdstrike_probe_sched_process_exec+0x4f5/0x6c0 [falcon_lsm_serviceable]
[77384.468770]  cshook_systemcalltable_post_fallocate+0xc3/0xe0 [falcon_lsm_serviceable]
[77384.468960]  unload_network_ops_symbols+0x873c/0xb560 [falcon_lsm_pinned_16303]
[77384.469151]  ? do_syscall_64+0x5b/0x1b0
[77384.469336]  ? entry_SYSCALL_64_after_hwframe+0x61/0xc6
[77384.469522] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_16303(E) vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock intel_rapl_msr intel_rapl_common intel_uncore_frequency_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper sd_mod t10_pi sg syscopyarea sysfillrect sysimgblt drm ahci ata_piix libahci crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi dm_mod fuse [last unloaded: falcon_kal]
[77384.470408] CR2: 0000073321736b48

• RHEL7 system crashed due to a general protection fault (GPF) in the mutex_lock() function with the following call traces.

[431403.536720] general protection fault: 0000 [#1] SMP 
[431403.536744] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) 
[431403.536984]  dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
[431403.537005] CPU: 2 PID: 18060 Comm: udsagent Kdump: loaded Tainted: POE  ------------  3.10.0-1160.76.1.el7.x86_64 #1
[431403.537030] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[431403.537059] task: ffff98f3cb0e9080 ti: ffff98f5b448c000 task.ti: ffff98f5b448c000
[431403.537076] RIP: 0010:[<ffffffff91d8a835>]  [<ffffffff91d8a835>] mutex_lock+0x15/0x2f
[431403.537100] RSP: 0018:ffff98f5b448fe50  EFLAGS: 00010246
[431403.537112] RAX: 0000000000000000 RBX: dead000000000128 RCX: ffff98f5b448ffd8
[431403.537129] RDX: 0000000000000000 RSI: 0000000000000002 RDI: dead000000000128
[431403.537145] RBP: ffff98f5b448fe58 R08: 00000000000001ff R09: 0000000180080004
[431403.537160] R10: 00000000eb3ce001 R11: ffff98f9eb3cd000 R12: 0000000000000001
[431403.537181] R13: ffff98f5b448ff08 R14: dead000000000128 R15: dead0000000000e8
[431403.537198] FS:  00007f51584b9700(0000) GS:ffff98fa7fc80000(0000) knlGS:0000000000000000
[431403.537216] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[431403.537229] CR2: 00007f48a488f000 CR3: 0000000819e1e000 CR4: 00000000007607e0
[431403.537274] PKRU: 55555554
[431403.537281] Call Trace:
[431403.537294]  [<ffffffffc07df82e>] 0xffffffffc07df82d
[431403.537310]  [<ffffffffc09e40e5>] crowdstrike_probe_sched_process_exec+0x4f5/0x6d0 [falcon_lsm_serviceable]
[431403.537332]  [<ffffffffc09e59e9>] cshook_systemcalltable_post_open+0xa9/0xc0 [falcon_lsm_serviceable]
[431403.537356]  [<ffffffffc07ac459>] unload_network_ops_symbols+0x6c89/0x7210 [falcon_lsm_pinned_14306]
[431403.537377]  [<ffffffff91d99f92>] system_call_fastpath+0x25/0x2a
[431403.537391] Code: ff 07 7f 05 e8 fd 0a 00 00 5d c3 0f 1f 00 83 ea 01 89 50 10 eb d9 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb e8 2e 1e 00 00 48 89 df <f0> ff 0f 79 05 e8 61 0b 00 00 65 48 8b 04 25 c0 0e 01 00 48 89 
[431403.537485] RIP  [<ffffffff91d8a835>] mutex_lock+0x15/0x2f
[431403.537499]  RSP <ffff98f5b448fe50>

• Another RHEL7 shows the function cskal_mutex_lock in the module falcon_kal instead.

crash> bt
PID: 10127    TASK: ffff9267fed0c200  CPU: 0    COMMAND: "mount.nfs"
 #0 [ffff92662b743b58] panic at ffffffff945ab837
 #1 [ffff92662b743bd8] oops_end at ffffffff945bc839
 #2 [ffff92662b743c00] no_context at ffffffff93e7970c
 #3 [ffff92662b743c50] __bad_area_nosemaphore at ffffffff93e799ea
 #4 [ffff92662b743ca0] bad_area_nosemaphore at ffffffff93e79b14
 #5 [ffff92662b743cb0] __do_page_fault at ffffffff945bf8d0
 #6 [ffff92662b743d20] do_page_fault at ffffffff945bfb05
 #7 [ffff92662b743d50] page_fault at ffffffff945bb7b8
    [exception RIP: mutex_lock+0x15]
    RIP: ffffffff945b6025  RSP: ffff92662b743e08  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: 00000001000000e6  RCX: ffff92662b743fd8
    RDX: 0000000000000000  RSI: 0000000000000005  RDI: 00000001000000e6
    RBP: ffff92662b743e10   R8: 00005612de19a220   R9: 0000000000000000
    R10: ffff926800a1f040  R11: fffff9dffbf38000  R12: 0000000000000001
    R13: ffff92662b743ee8  R14: 00000001000000e6  R15: 00000001000000a6
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff92662b743e18] cskal_mutex_lock at ffffffffc06ddc1e [falcon_kal]
 #9 [ffff92662b743e28] crowdstrike_probe_sched_process_exec at ffffffffc0b2e405 [falcon_lsm_serviceable]
#10 [ffff92662b743e68] cshook_systemcalltable_post_mount at ffffffffc0b2fc2f [falcon_lsm_serviceable]
#11 [ffff92662b743ed0] unload_network_ops_symbols at ffffffffc0739149 [falcon_lsm_pinned_15402]
#12 [ffff92662b743f50] system_call_fastpath at ffffffff945c539a
    RIP: 00007fe06caac26a  RSP: 00007ffec004e4e8  RFLAGS: 00000246
    RAX: 00000000000000a5  RBX: 0000000000000000  RCX: ffffffffffffffff
    RDX: 00005612de19a220  RSI: 00005612de19a240  RDI: 00005612de19a350
    RBP: 00007ffec004e750   R8: 00005612de19a950   R9: 00005612de19a950
    R10: 0000000000000000  R11: 0000000000000246  R12: 00007ffec004e750
    R13: 00005612de19a590  R14: 0000000000000010  R15: 00007ffec004e640
    ORIG_RAX: 00000000000000a5  CS: 0033  SS: 002b

• RHEL6 system crashed due to a NULL pointer dereference in the mutex_lock() function with the following call traces.

BUG: unable to handle kernel NULL pointer dereference at 00000000000000d9
IP: [<ffffffff8155c201>] mutex_lock+0x21/0x50
Kernel PGD 0 
User   PGD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 1 
Modules linked in: falcon_lsm_serviceable(P)(U) falcon_nf_netcontain(P)(U) falcon_kal(U) 
Pid: 3121, comm: mount.nfs Tainted: P  -------------- 2.6.32-754.48.1.el6.x86_64 #1 HP ProLiant DL380 G7
RIP: 0010:[<ffffffff8155c201>]  [<ffffffff8155c201>] mutex_lock+0x21/0x50
RSP: 0018:ffff8823f5eabe00  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000000000000d9 RCX: ffff8823f5eabef8
RDX: 0000000000000000 RSI: 0000000000000002 RDI: 00000000000000d9
RBP: ffff8823f5eabe10 R08: 0000559700cc643b R09: 0000000000000003
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8823f5eabef8 R14: 00000000000000d9 R15: 0000000000000099
FS:  00007f220165b700(0000) GS:ffff88125f800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000d9 CR3: 00000023f2dc6000 CR4: 00000000000207e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mount.nfs (pid: 3121, threadinfo ffff8823f5ea8000, task ffff88240059d520)
Stack:
 ffff8823eeaf1850 0000000000000001 ffff8823f5eabe20 ffffffffa02d966e
<d> ffff8823f5eabe60 ffffffffa0820965 0000000000000003 ffffffffa08d23e0
<d> 00005597029b2ea0 0000559700cc643b 0000000000000003 ffff8823f5eabef8
Call Trace:
 [<ffffffffa02d966e>] 0xffffffffa02d966e
 [<ffffffffa0820965>] crowdstrike_probe_sched_process_exec+0x4f5/0x6d0 [falcon_lsm_serviceable]
 [<ffffffffa082219f>] cshook_systemcalltable_post_mount+0xcf/0xf0 [falcon_lsm_serviceable]
 [<ffffffff811c85c8>] ? sys_mount+0xb8/0xe0
 [<ffffffffa03bb819>] load_network_ops_symbols+0x5339/0x6110 [falcon_lsm_pinned_13804]
 [<ffffffff810f410e>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff815663a7>] system_call_fastpath+0x35/0x3a
Code: c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c 89 64 24 08 0f 1f 44 00 00 48 89 fb e8 62 f5 ff ff 48 89 df <f0> ff 0f 79 05 e8 e5 06 00 00 65 48 8b 04 25 08 fc 00 00 48 2d 
RIP  [<ffffffff8155c201>] mutex_lock+0x21/0x50
 RSP <ffff8823f5eabe00>
CR2: 00000000000000d9