Server crashes with "BUG: unable to handle kernel paging request at ffffb567ad9a82e0 in the code path of third party module "falcon_lsm_serviceable"

Diagnostic Steps

• Server crashed due to [falcon_lsm_serviceable] module function as shown below,

Kernel ring buffer:

crash> log
[420181.203210] BUG: unable to handle kernel paging request at ffffb567ad9a82e0
[420181.235643] IP: [<ffffffffa9bb7115>] down_read+0x15/0x40
[420181.260781] PGD 17fd4e067 PUD 3fffc76067 PMD 0 
[420181.282362] Oops: 0002 [#1] SMP 
[420181.297878] Modules linked in: tcp_diag inet_diag falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15003(E) nfsv3 nfs_acl nfs lockd grace fscache team_mode_activebackup team ext4 mbcache jbd2 rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm sb_edac intel_powerclamp i2c_algo_bit ttm coretemp drm_kms_helper syscopyarea intel_rapl sysfillrect sysimgblt fb_sys_fops iosf_mbi drm drm_panel_orientation_quirks iTCO_wdt iTCO_vendor_support ocrdma(T) kvm_intel kvm ib_core i2c_i801 pcspkr sg irqbypass ioatdma hpilo crc32_pclmul ghash_clmulni_intel aesni_intel hpwdt lrw gf128mul glue_helper lpc_ich ablk_helper dca cryptd ipmi_si ipmi_devintf
[420181.630482]  wmi ipmi_msghandler acpi_power_meter dm_multipath binfmt_misc ip_tables xfs libcrc32c dm_snapshot dm_bufio sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common serio_raw crc32c_intel hpsa be2net scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod fuse
[420181.743406] CPU: 13 PID: 7159 Comm: rpm Kdump: loaded Tainted: P            E  ------------ T 3.10.0-1160.90.1.el7.x86_64 #1
[420181.796556] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 08/04/2022
[420181.835855] task: ffff945971156300 ti: ffff947b2bb7c000 task.ti: ffff947b2bb7c000
[420181.871486] RIP: 0010:[<ffffffffa9bb7115>]  [<ffffffffa9bb7115>] down_read+0x15/0x40       <<-------
[420181.909215] RSP: 0018:ffff947b2bb7fd08  EFLAGS: 00010246
[420181.934626] RAX: ffffb567ad9a82e0 RBX: ffffb567ad9a82e0 RCX: ffff947b2bb7ffd8
[420181.969766] RDX: 0000000000000000 RSI: ffffb567ad9a82d0 RDI: ffffb567ad9a82e0
[420182.003651] RBP: ffff947b2bb7fd10 R08: ffff947b2bb7fd54 R09: ffff946427f45b38
[420182.037634] R10: 00007f36b34cfc8c R11: 0000000000000246 R12: ffff947b2bb7fdc8
[420182.071511] R13: ffff946427f45b00 R14: 00007f36b5809ced R15: ffff947b2bb7ff10
[420182.105178] FS:  00007f36b5e90840(0000) GS:ffff949bbfc80000(0000) knlGS:0000000000000000
[420182.143815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[420182.170891] CR2: ffffb567ad9a82e0 CR3: 0000007fe49d4000 CR4: 00000000003607e0
[420182.204381] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[420182.238102] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[420182.272027] Call Trace:
[420182.283943]  [<ffffffffc09be8ae>] 0xffffffffc09be8ad
[420182.307804]  [<ffffffffc0a16260>] cshook_security_inode_free_security+0x3110/0x5700 [falcon_lsm_serviceable]
[420182.354873]  [<ffffffffc0a96618>] cshook_network_ops_inet6_sockraw_recvmsg+0x19f88/0x23970 [falcon_lsm_serviceable]
[420182.403701]  [<ffffffffc0a11f76>] cshook_security_file_free_security+0x4806/0x59e0 [falcon_lsm_serviceable]
[420182.451124]  [<ffffffffc0a10893>] cshook_security_file_free_security+0x3123/0x59e0 [falcon_lsm_serviceable]
[420182.496335]  [<ffffffffc0a13d88>] cshook_security_inode_free_security+0xc38/0x5700 [falcon_lsm_serviceable]
[420182.542742]  [<ffffffffc0a9ba13>] cshook_network_ops_inet6_sockraw_recvmsg+0x1f383/0x23970 [falcon_lsm_serviceable]
[420182.594982]  [<ffffffffc0a499ae>] cshook_systemcalltable_pre_write+0x2e/0x30 [falcon_lsm_serviceable]
[420182.641804]  [<ffffffffc09ab676>] unload_network_ops_symbols+0x8cb6/0x8e80 [falcon_lsm_pinned_15003]
[420182.686420]  [<ffffffffa9bc539a>] system_call_fastpath+0x25/0x2a
[420182.715737] Code: 00 00 00 31 f6 48 89 e5 e8 69 fe ff ff 5d c3 cc cc cc cc 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb e8 fe 0d 00 00 48 89 d8 <f0> 48 ff 00 79 05 e8 b0 72 bf ff 65 48 8b 04 25 c0 0e 01 00 48 
[420182.808121] RIP  [<ffffffffa9bb7115>] down_read+0x15/0x40
[420182.835184]  RSP <ffff947b2bb7fd08>
[420182.852841] CR2: ffffb567ad9a82e0

Backtrace of panic task:

• The server panicked within the function down_read().

crash> bt
PID: 7159     TASK: ffff945971156300  CPU: 13   COMMAND: "rpm"
 #0 [ffff947b2bb7f990] machine_kexec at ffffffffa9469514
 #1 [ffff947b2bb7f9f0] __crash_kexec at ffffffffa9529e82
 #2 [ffff947b2bb7fac0] crash_kexec at ffffffffa9529f78
 #3 [ffff947b2bb7fad8] oops_end at ffffffffa9bbc818
 #4 [ffff947b2bb7fb00] no_context at ffffffffa947974c
 #5 [ffff947b2bb7fb50] __bad_area_nosemaphore at ffffffffa9479a2a
 #6 [ffff947b2bb7fba0] bad_area_nosemaphore at ffffffffa9479b54
 #7 [ffff947b2bb7fbb0] __do_page_fault at ffffffffa9bbf8d0
 #8 [ffff947b2bb7fc20] do_page_fault at ffffffffa9bbfb05
 #9 [ffff947b2bb7fc50] page_fault at ffffffffa9bbb7b8
    [exception RIP: down_read+21]                               <<---------
    RIP: ffffffffa9bb7115  RSP: ffff947b2bb7fd08  RFLAGS: 00010246
    RAX: ffffb567ad9a82e0  RBX: ffffb567ad9a82e0  RCX: ffff947b2bb7ffd8
    RDX: 0000000000000000  RSI: ffffb567ad9a82d0  RDI: ffffb567ad9a82e0
    RBP: ffff947b2bb7fd10   R8: ffff947b2bb7fd54   R9: ffff946427f45b38
    R10: 00007f36b34cfc8c  R11: 0000000000000246  R12: ffff947b2bb7fdc8
    R13: ffff946427f45b00  R14: 00007f36b5809ced  R15: ffff947b2bb7ff10
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff947b2bb7fd18] cskal_down_read at ffffffffc09be8ae [falcon_kal]
#11 [ffff947b2bb7fd28] cshook_security_inode_free_security at ffffffffc0a16260 [falcon_lsm_serviceable]
#12 [ffff947b2bb7fd38] cshook_network_ops_inet6_sockraw_recvmsg at ffffffffc0a96618 [falcon_lsm_serviceable]
#13 [ffff947b2bb7fda8] cshook_security_file_free_security at ffffffffc0a11f76 [falcon_lsm_serviceable]
    RIP: 00007f36b31f6ba0  RSP: 00007ffc89d4c844  RFLAGS: 00000296
    RAX: 0000000000000001  RBX: 0000000000000007  RCX: 000000006d3e202d
    RDX: 0000000000000007  RSI: 00007f36b5809ced  RDI: 0000000000000002
    RBP: 00007f36b5809ced   R8: 00007f36b34d09f0   R9: 0000000000000000
    R10: 00007f36b34cfc8c  R11: 0000000000000246  R12: 00007f36b34cf1c0
    R13: 0000000000000007  R14: 0000000000000d70  R15: 00007f36b34ca838
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash> 

Dis-assembly of exception RIP:

• The panic occurred while dereferencing the virtual address stored in the register %rax.

crash> dis -rl ffffffffa9bb7115 |tail
0xffffffffa9bb7100 <down_read>: nopl   0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffa9bb7105 <down_read+5>:   push   %rbp
0xffffffffa9bb7106 <down_read+6>:   mov    %rsp,%rbp
0xffffffffa9bb7109 <down_read+9>:   push   %rbx
0xffffffffa9bb710a <down_read+10>:  mov    %rdi,%rbx
/usr/src/debug/kernel-3.10.0-1160.90.1.el7/linux-3.10.0-1160.90.1.el7.x86_64/kernel/rwsem.c: 21
0xffffffffa9bb710d <down_read+13>:  call   0xffffffffa9bb7f10 <_cond_resched>
/usr/src/debug/kernel-3.10.0-1160.90.1.el7/linux-3.10.0-1160.90.1.el7.x86_64/arch/x86/include/asm/rwsem.h: 65
0xffffffffa9bb7112 <down_read+18>:  mov    %rbx,%rax
0xffffffffa9bb7115 <down_read+21>:  lock incq (%rax)
                                                  |
                                                  --------
                                                         |
                                                         V
 BUG: unable to handle kernel paging request at ffffb567ad9a82e0

• Checking further the value in %rax was populated at <cshook_security_inode_free_security+12551> by the function provided by unsigned kernel module [falcon_lsm_serviceable].

crash> dis -rl ffffffffc0a16260
[--]
0xffffffffc0a1624f <cshook_security_inode_free_security+12543>: imul   0x24(%rdi),%esi
0xffffffffc0a16253 <cshook_security_inode_free_security+12547>: add    0x18(%rdi),%rsi
0xffffffffc0a16257 <cshook_security_inode_free_security+12551>: lea    0x10(%rsi),%rdi
0xffffffffc0a1625b <cshook_security_inode_free_security+12555>: call   0xffffffffc09be8a0 <cskal_down_read>
0xffffffffc0a16260 <cshook_security_inode_free_security+12560>: xor    %eax,%eax
[--]


crash> px (0xffffb567ad9a82d0+0x10)
$1 = 0xffffb567ad9a82e0
                    |
                    --------------------------------------
                                                         |
                                                         V

"BUG: unable to handle kernel paging request at ffffb567ad9a82e0"

crash> sym cshook_security_inode_free_security
ffffffffc0a13150 (t) cshook_security_inode_free_security [falcon_lsm_serviceable] 
                          ^                                           ^
                          |                                           |
  [ Function within the module code ]                         [ Module Name ]

Third-party modules:

• Details of an unsigned (U) module: [falcon_lsm_serviceable].

crash> mod -t
NAME                     TAINTS
falcon_nf_netcontain     PE
falcon_lsm_pinned_15003  E
falcon_kal               E
falcon_lsm_serviceable   PE   <<------
ocrdma                   T
crash>