Kernel panic due to third party kernel module "protection_agent"

Issue

• Kernel panicked with either of the following entries:

  • Null pointer dereference where stack trace refers to third party kernel module:

  BUG: unable to handle kernel NULL pointer dereference at           (null)
  ...
  Call Trace:
  [<ffffffffc06b0c07>] ? CrusherCompareFunc+0x37/0x140 [protection_agent]
  [<ffffffffc06b0bd0>] ? levenshtein_fast+0x70/0x70 [protection_agent]
  [<ffffffffc06b0983>] pid_list_search_if+0x33/0x60 [protection_agent]
  

  • Page Table Corruption, caused by exception in radix_tree_lookup instruction pointer:

  [10114747.695531] BUG: unable to handle kernel paging request at ffffa77819234c20
  [10114747.695562] IP: [<ffffffff937a344c>] __radix_tree_lookup+0x2c/0x100
  ...
  [10114747.695605] Modules linked in: protection_agent(OE) joydev unix_diag af_packet_diag
  ...
  [10114747.706808] Call Trace:
  [10114747.707385]  [<ffffffff937a3542>] radix_tree_lookup_slot+0x22/0x50
  [10114747.707972]  [<ffffffff935c83de>] __find_get_page+0x1e/0xa0
  

  • fatal page fault in the __task_pid_nr_ns() routine called from the NNS_CHANNEL_NEXT_SendData()→init_ids() functions:

PID: 1390359  TASK: ffff947c857c8000  CPU: 1    COMMAND: "task"
…
 #6 [ffffa1d300514ce0] asm_exc_page_fault at ffffffffb7400bb2
    [exception RIP: __task_pid_nr_ns+52]
    RIP: ffffffffb673b414  RSP: ffffa1d300514d90  RFLAGS: 00010286
    RAX: ffff947d83306d00  RBX: 0000000000000000  RCX: 0000000000000001
    RDX: ffff947c857c8000  RSI: 0000000000000002  RDI: ffff947c857c8000
    RBP: ffff947daa000fc8   R8: 0000000000000060   R9: 0000000000000000
    R10: ffffa1d300514db0  R11: 000000000000000f  R12: 0000000000000000
    R13: 0000000000000032  R14: ffffa1d300514e2f  R15: ffff947da6fedc60
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffa1d300514da8] init_ids at ffffffffc0949495 [protection_agent]
 #8 [ffffa1d300514db8] NNS_CHANNEL_NEXT_SendData at ffffffffc09841fb [protection_agent]
 #9 [ffffa1d300514e20] NNS_STRMR_MP_OpenConnectionSendMessage at ffffffffc0983cdd [protection_agent]
#10 [ffffa1d300514e48] NNS_STRMR_MP_OpenConnectionManage at ffffffffc098404a [protection_agent]
#11 [ffffa1d300514e88] NNS_FLOW_SESSION_SessionAgeTimeout at ffffffffc0986478 [protection_agent]
#12 [ffffa1d300514ec0] kmod_nnsx_timer_callback at ffffffffc095033e [protection_agent]
#13 [ffffa1d300514ec8] call_timer_fn at ffffffffb67e754a
#14 [ffffa1d300514ee8] __run_timers at ffffffffb67e7834
#15 [ffffa1d300514f80] run_timer_softirq at ffffffffb67e7906
…

• general protection fault in the kfree() routine called from the NNS_FLOW_SESSION_SessionFree() function:

crash> bt
PID: 94305    TASK: ffffa03dd7bc22c0  CPU: 1    COMMAND: "task"
…
 #4 [ffffbcb34a2939d0] asm_exc_general_protection at ffffffff84e00b32
    [exception RIP: kfree+0x5c]
    RIP: ffffffff8457dc3c  RSP: ffffbcb34a293a88  RFLAGS: 00010207
    RAX: 00d3eddc4dcc00c0  RBX: 3500300033003100  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffffffffc1487e8c  RDI: 3500300033003100
    RBP: ffffa0532b54ebee   R8: 0000000000000000   R9: 000000000000000f
    R10: 000000000000000f  R11: ffffa0469d29ba1f  R12: 35003000b3003100
    R13: ffffa03dd12c0000  R14: 0000000000000001  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #5 [ffffbcb34a293ac0] NNS_FLOW_SESSION_SessionFree at ffffffffc1487e8c [protection_agent]
 #6 [ffffbcb34a293ad0] NNS_FW_HT_INTER_ElementBucketListFree at ffffffffc147d49f [protection_agent]
 #7 [ffffbcb34a293b08] NNS_FW_HT_CloseEHandle at ffffffffc1482c71 [protection_agent]
 #8 [ffffbcb34a293b80] NNS_FLOW_SESSION_SessionListRelease at ffffffffc1488a2b [protection_agent]
 #9 [ffffbcb34a293b90] NNS_FLOW_ProcessFlowEstablish at ffffffffc14839b1 [protection_agent]
#10 [ffffbcb34a293be8] avpm_simulate_flow_stablished.constprop.0 at ffffffffc1454b84 [protection_agent]
#11 [ffffbcb34a293c88] avpm_event_socket_data at ffffffffc14551d3 [protection_agent]
#12 [ffffbcb34a293d20] kauth_socket_send_hook at ffffffffc1449762 [protection_agent]
#13 [ffffbcb34a293d48] sock_sendmsg at ffffffff84ac2da6
#14 [ffffbcb34a293d60] __sys_sendto at ffffffff84ac4950
#15 [ffffbcb34a293e90] __x64_sys_sendto at ffffffff84ac49e0
#16 [ffffbcb34a293e98] do_syscall_64 at ffffffff84d2f15c
…

• triggering the BUG_ON() assertion in the __slab_free() routine called from the NCGP_PoolFreeEntry_TC() function:

crash> bt
PID: 20918    TASK: ffff888bd19f0000  CPU: 4    COMMAND: "task"
…
 #6 [ffff9a5b8c8cf950] asm_exc_invalid_op at ffffffffa0400b86
    [exception RIP: __slab_free+0x225]
    RIP: ffffffff9fb7da25  RSP: ffff9a5b8c8cfa00  RFLAGS: 00010246
    RAX: ffff888b68b04100  RBX: ffff888b68b04000  RCX: 0000000080200018
    RDX: 0000000080200018  RSI: 0000000000000001  RDI: ffff888900042a00
    RBP: ffffcc38cda2c100   R8: 0000000000000001   R9: 0000000000000000
    R10: ffff888c6ca39000  R11: 0000000000000000  R12: ffff888b68b04000
    R13: ffff888900042a00  R14: ffff888b68b04000  R15: 0000000080200018
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff9a5b8c8cfa90] NCGP_PoolFreeEntry_TC at ffffffffc11aa6ea [protection_agent]
 #8 [ffff9a5b8c8cfaa0] NNS_FLOW_SESSION_SessionFree at ffffffffc11dbed0 [protection_agent]
 #9 [ffff9a5b8c8cfab0] NNS_FW_HT_INTER_ElementBucketListFree at ffffffffc11d149f [protection_agent]
#10 [ffff9a5b8c8cfae8] NNS_FW_HT_CloseEHandle at ffffffffc11d6c71 [protection_agent]
#11 [ffff9a5b8c8cfb60] NNS_FLOW_SESSION_SessionListRelease at ffffffffc11dca2b [protection_agent]
#12 [ffff9a5b8c8cfb70] NNS_FLOW_ProcessFlowEstablish at ffffffffc11d79b1 [protection_agent]
#13 [ffff9a5b8c8cfbc8] avpm_simulate_flow_stablished.constprop.0 at ffffffffc11a8b84 [protection_agent]
#14 [ffff9a5b8c8cfc68] avpm_event_socket_data at ffffffffc11a91d3 [protection_agent]
#15 [ffff9a5b8c8cfd00] kauth_socket_send_hook at ffffffffc119d762 [protection_agent]
#16 [ffff9a5b8c8cfd28] sock_sendmsg at ffffffffa00c2da6
#17 [ffff9a5b8c8cfd40] __sys_sendto at ffffffffa00c4950
#18 [ffff9a5b8c8cfe70] __x64_sys_sendto at ffffffffa00c49e0
…