Kernel crash at __memmove+0x56/0x1b0 function

Diagnostic Steps

System information:

        CPUS: 4
        DATE: Wed Dec  6 12:35:28 EST 2023
      UPTIME: 04:26:05
LOAD AVERAGE: 5.56, 4.95, 4.83
       TASKS: 1077
    NODENAME: localhost
     RELEASE: 4.18.0-477.27.1.el8_8.x86_64
     VERSION: #1 SMP Thu Aug 31 10:29:22 EDT 2023
     MACHINE: x86_64  (2593 Mhz)
      MEMORY: 8 GB
       PANIC: "BUG: unable to handle kernel paging request at ffff9e6fd31f2000"

System Hardware information:

crash> sys -i | head -n5
        DMI_BIOS_VENDOR: VMware, Inc.
       DMI_BIOS_VERSION: VMW71.00V.18227214.B64.2106252220
          DMI_BIOS_DATE: 06/25/2021
         DMI_SYS_VENDOR: VMware, Inc.
       DMI_PRODUCT_NAME: VMware7,1

Kernel Ring Buffer:

crash> log
[..]
[15964.881928] BUG: unable to handle kernel paging request at ffff9e6fd31f2000
[15964.882061] PGD 18d803067 P4D 18d803067 PUD 100034063 PMD 1a9df1063 PTE 80000001931f2161
[15964.882085] Oops: 0003 [#1] SMP PTI
[15964.882097] CPU: 2 PID: 8608 Comm: .NET ThreadPool Kdump: loaded Tainted: P           O     --------- -  - 4.18.0-477.27.1.el8_8.x86_64 #1
[15964.882141] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
[15964.882184] RIP: 0010:__memmove+0x56/0x1b0
[15964.882207] Code: 00 72 05 40 38 fe 74 3c 48 83 ea 20 48 83 ea 20 4c 8b 1e 4c 8b 56 08 4c 8b 4e 10 4c 8b 46 18 48 8d 76 20 4c 89 1f 4c 89 57 08 <4c> 89 4f 10 4c 89 47 18 48 8d 7f 20 73 d4 48 83 c2 20 e9 a9 00 00
[15964.882292] RSP: 0018:ffffae7146603d38 EFLAGS: 00010283
[15964.882309] RAX: ffff9e6fd31f100e RBX: ffff9e6fd31f1000 RCX: 0000000000000ff1
[15964.882334] RDX: ffffffffffffffef RSI: ffff9e6fd31f201c RDI: ffff9e6fd31f1fee
[15964.882363] RBP: ffff9e6fd31f100e R08: 725720646e612064 R09: 616552202e652e69
[15964.882386] R10: 20746c7561666564 R11: 206f7420442e3630 R12: ffff9e6fd31f101c
[15964.882405] R13: ffffae7146603d98 R14: ffffae7146603da0 R15: 0000000000000000
[15964.882433] FS:  00007f69204a9b30(0000) GS:ffff9e7075e80000(0000) knlGS:0000000000000000
[15964.882462] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[15964.882477] CR2: ffff9e6fd31f2000 CR3: 00000001e2758006 CR4: 00000000003706e0
[15964.882524] Call Trace:
[15964.882562]  mfe_aac_get_file_path+0xfc/0x120 [mfe_aac_1007152489]
[15964.882596]  mfe_aac_extract_path+0xc8/0xe0 [mfe_aac_1007152489]
[15964.882628]  mfe_aac_sys_open_64_bit+0x99/0x280 [mfe_aac_1007152489]
[15964.882653]  ? seccomp_run_filters+0x77/0x130
[15964.882672]  ? __handle_mm_fault+0x453/0x6c0
[15964.882694]  ? __seccomp_filter+0x3e/0x4b0
[15964.882715]  ? __audit_syscall_entry+0xf2/0x140
[15964.882733]  ? syscall_trace_enter+0x1ff/0x2d0
[15964.882752]  ? mfe_fileaccess_sys_open_64_bit+0x34/0x1f0 [mfe_fileaccess_1007152489]
[15964.883536]  mfe_fileaccess_sys_open_64_bit+0x34/0x1f0 [mfe_fileaccess_1007152489]
[15964.884188]  do_syscall_64+0x5b/0x1b0
[15964.884733]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[15964.885271] RIP: 0033:0x7f69bd379f63
[15964.885801] Code: c3 8b 07 85 c0 75 24 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> e9 70 d0 ff ff 41 54 b8 02 00 00 00 55 48 89 f5 be 00 88 08 00
[15964.887096] RSP: 002b:00007f69204a8a58 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[15964.887672] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f69bd379f63
[15964.888307] RDX: 00000000000001b6 RSI: 0000000000088041 RDI: 00007f6921add150
[15964.888853] RBP: 00007f69204a9b30 R08: 0000000000000000 R09: 0000000000000000
[15964.889486] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6922fb9520
[15964.890052] R13: 000000000002c4c0 R14: 00000000000001b6 R15: 00007f6921add150
[15964.890635] Modules linked in: nfnetlink_queue ipt_REJECT nf_reject_ipv4 xt_comment xt_NFQUEUE xt_REDIRECT seqiv esp4 nfsv3 nfs_acl mfe_fileaccess_1007152489(O) ip_vs_rr xt_ipvs ip_vs binfmt_misc b9k_87163(PO) cbproxy_cbp_8716_20230531(PO) xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_policy xt_mark xt_bpf xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc mfe_aac_1007152489(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common sb_edac crct10dif_pclmul crc32_pclmul vmw_balloon ghash_clmulni_intel rapl vfat fat joydev pcspkr vmw_vmci i2c_piix4 xfs libcrc32c sr_mod cdrom ata_generic vmwgfx drm_ttm_helper ttm drm_kms_helper ahci sd_mod t10_pi syscopyarea sg libahci
[15964.890716]  sysfillrect sysimgblt fb_sys_fops ata_piix drm crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
[15964.897759] Red Hat flags: eBPF/event
[15964.898638] CR2: ffff9e6fd31f2000
[..]

• The panic task is '.NET ThreadPool' PID (8608):

crash> set -p
    PID: 8608
COMMAND: ".NET ThreadPool"
   TASK: ffff9e70260b2800  [THREAD_INFO: ffff9e70260b2800]
    CPU: 2
  STATE: TASK_RUNNING (PANIC)

Backtrace of the panic task:

crash> bt
PID: 8608     TASK: ffff9e70260b2800  CPU: 2    COMMAND: ".NET ThreadPool"
 #0 [ffffae7146603a60] machine_kexec at ffffffff9586c1f3
 #1 [ffffae7146603ab8] __crash_kexec at ffffffff959b59aa
 #2 [ffffae7146603b78] crash_kexec at ffffffff959b68e1
 #3 [ffffae7146603b90] oops_end at ffffffff9582a9c1
 #4 [ffffae7146603bb0] no_context at ffffffff9587e913
 #5 [ffffae7146603c08] __bad_area_nosemaphore at ffffffff9587ec8c
 #6 [ffffae7146603c50] do_page_fault at ffffffff9587f8a7
 #7 [ffffae7146603c80] page_fault at ffffffff9640116e
    [exception RIP: memmove+86]
    RIP: ffffffff961ed3a6  RSP: ffffae7146603d38  RFLAGS: 00010283
    RAX: ffff9e6fd31f100e  RBX: ffff9e6fd31f1000  RCX: 0000000000000ff1
    RDX: ffffffffffffffef  RSI: ffff9e6fd31f201c  RDI: ffff9e6fd31f1fee
    RBP: ffff9e6fd31f100e   R8: 725720646e612064   R9: 616552202e652e69
    R10: 20746c7561666564  R11: 206f7420442e3630  R12: ffff9e6fd31f101c
    R13: ffffae7146603d98  R14: ffffae7146603da0  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffae7146603d38] mfe_aac_get_file_path at ffffffffc0a0f24c [mfe_aac_1007152489]
 #9 [ffffae7146603d58] mfe_aac_extract_path at ffffffffc0a0f478 [mfe_aac_1007152489]
#10 [ffffae7146603d88] mfe_aac_sys_open_64_bit at ffffffffc0a091d9 [mfe_aac_1007152489]
#11 [ffffae7146603f00] mfe_fileaccess_sys_open_64_bit at ffffffffc0e1e1b4 [mfe_fileaccess_1007152489]
#12 [ffffae7146603f38] do_syscall_64 at ffffffff958052fb
#13 [ffffae7146603f50] entry_SYSCALL_64_after_hwframe at ffffffff964000a9
    RIP: 00007f69bd379f63  RSP: 00007f69204a8a58  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000002  RCX: 00007f69bd379f63
    RDX: 00000000000001b6  RSI: 0000000000088041  RDI: 00007f6921add150
    RBP: 00007f69204a9b30   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000246  R12: 00007f6922fb9520
    R13: 000000000002c4c0  R14: 00000000000001b6  R15: 00007f6921add150
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b

Note:That we traverse through [mfe_aac_1007152489] code path, functions.

Dis-assembly:

crash> dis -rl ffffffff961ed3a6 | tail
/usr/src/debug/kernel-4.18.0-477.27.1.el8_8/linux-4.18.0-477.27.1.el8_8.x86_64/arch/x86/lib/memmove_64.S: 69
0xffffffff961ed397 <memmove+71>:    mov    0x18(%rsi),%r8
/usr/src/debug/kernel-4.18.0-477.27.1.el8_8/linux-4.18.0-477.27.1.el8_8.x86_64/arch/x86/lib/memmove_64.S: 70
0xffffffff961ed39b <memmove+75>:    lea    0x20(%rsi),%rsi
/usr/src/debug/kernel-4.18.0-477.27.1.el8_8/linux-4.18.0-477.27.1.el8_8.x86_64/arch/x86/lib/memmove_64.S: 72
0xffffffff961ed39f <memmove+79>:    mov    %r11,(%rdi)    <<---
/usr/src/debug/kernel-4.18.0-477.27.1.el8_8/linux-4.18.0-477.27.1.el8_8.x86_64/arch/x86/lib/memmove_64.S: 73
0xffffffff961ed3a2 <memmove+82>:    mov    %r10,0x8(%rdi)  <<---
/usr/src/debug/kernel-4.18.0-477.27.1.el8_8/linux-4.18.0-477.27.1.el8_8.x86_64/arch/x86/lib/memmove_64.S: 74
0xffffffff961ed3a6 <memmove+86>:    mov    %r9,0x10(%rdi)  <<----

crash> px (0xffff9e6fd31f1fee+0x10)
$7 = 0xffff9e6fd31f1ffe

crash> px (0xffff9e6fd31f1ffe+0x8)
$8 = 0xffff9e6fd31f2006

BUG: unable to handle kernel paging request at ffff9e6fd31f2000

• There were three consecutive accesses to the memory pointed by the address stored in the %rdi CPU register:

 71 
 72         movq %r11, 0*8(%rdi)
 73         movq %r10, 1*8(%rdi)
 74         movq %r9,  2*8(%rdi)

• The last one failed because it crossed a page boundary with a different protection against writes:

crash> log
[--]
[15964.882085] Oops: 0003 [#1] SMP PTI
[--]

crash> eval -b 0x3
hexadecimal: 3  
    decimal: 3  
      octal: 3
     binary: 0000000000000000000000000000000000000000000000000000000000000011
   bits set: 1 0 

Kernel Source arch/x86/include/asm/trap_pf.h

  8  *   bit 0 ==    0: no page found       1: protection fault
  9  *   bit 1  ==    0: read access             1: write access

• Dis-assembly mfe_aac_get_file_path function:

crash> dis -rl ffffffffc0a0f24c | tail
0xffffffffc0a0f22c <mfe_aac_get_file_path+220>: test   %rax,%rax
0xffffffffc0a0f22f <mfe_aac_get_file_path+223>: je     0xffffffffc0a0f24c <mfe_aac_get_file_path+252>
0xffffffffc0a0f231 <mfe_aac_get_file_path+225>: lea    0x1(%rax),%r12
0xffffffffc0a0f235 <mfe_aac_get_file_path+229>: mov    %r12,%rdi
0xffffffffc0a0f238 <mfe_aac_get_file_path+232>: call   0xffffffff961e1380 <strlen>
0xffffffffc0a0f23d <mfe_aac_get_file_path+237>: mov    %r12,%rsi
0xffffffffc0a0f240 <mfe_aac_get_file_path+240>: mov    %rbp,%rdi
0xffffffffc0a0f243 <mfe_aac_get_file_path+243>: lea    0x1(%rax),%rdx
0xffffffffc0a0f247 <mfe_aac_get_file_path+247>: call   0xffffffff961ed350 <memmove>
0xffffffffc0a0f24c <mfe_aac_get_file_path+252>: mov    %rbx,%rax

• The panic occurred while dereferencing the virtual address stored in the register %rdi. While checking further the value in
  %rdi was populated at <mfe_aac_get_file_path+252> by the function provided by Unsigned kernel module [mfe_aac_1007152489].

Third-Party Module

• The function mfe_aac_get_file_path() is part of an unsigned (U) module [mfe_aac_1007152489].

crash> sym mfe_aac_get_file_path
ffffffffc0a0f150 (t) mfe_aac_get_file_path [mfe_aac_1007152489] 
                          ^                            ^
                          |                            |
  [ Function within the module code ]           [ Module Name ]

crash> mod -t
NAME                       TAINTS
mfe_aac_1007152489         O    <<<-----
cbproxy_cbp_8716_20230531  PO
b9k_87163                  PO
mfe_fileaccess_1007152489  O    

crash>  mod | grep -e NAME -e mfe_aac_1007152489
     MODULE       NAME                                     BASE           SIZE  OBJECT FILE
ffffffffc0a2c100  mfe_aac_1007152489                 ffffffffc0a06000   192512  (not loaded)  [CONFIG_KALLSYMS]

crash> module.state,name,version,srcversion,sig_ok ffffffffc0a2c100
  state = MODULE_STATE_LIVE,
  name = "mfe_aac_1007152489\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
  version = 0x0,
  srcversion = 0xffff9e6f57097ee0 "46EF770B349303F6D3B3EAD",
  sig_ok = true,