NULL pointer dereference in memset() function with probe_syscalls 3rd party module

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • probe_syscalls kernel module

Issue

Panic in memset() function with probe_syscalls module and memory fragmentation

Resolution

Unload and/or blocklisting the probe_syscalls kernel module.

Contact the vendor of the probe_syscalls module as it is not supported by Red Hat directly.

Root Cause

The panic happens in the memset() function due to a NULL pointer value in the %rdi CPU register after get_current_proc_path.constprop.10 calls kmalloc_order_trace which fails to allocate a page due to memory fragmentation.

Diagnostic Steps

Pre-requisites

  1. Deploy kdump in Order to Collect a vmcore:

  2. Prepare crash Environment for vmcore Analysis:

Vmcore Analysis

  1. Backtrace of the panic task shows memset() called from the get_current_proc_path.constprop.10 function and %rdi is 0x0:

    PID: 2775     TASK: ffff9ce05771e180  CPU: 7    COMMAND: "in:imjournal"
 #0 [ffff9ce03ecbbb50] machine_kexec at ffffffff9f6640d4
 #1 [ffff9ce03ecbbbb0] __crash_kexec at ffffffff9f71e822
 #2 [ffff9ce03ecbbc80] crash_kexec at ffffffff9f71e910
 #3 [ffff9ce03ecbbc98] oops_end at ffffffff9fd71798
 #4 [ffff9ce03ecbbcc0] no_context at ffffffff9fd5feb9
 #5 [ffff9ce03ecbbd10] __bad_area_nosemaphore at ffffffff9fd5ff50
 #6 [ffff9ce03ecbbd60] bad_area_nosemaphore at ffffffff9fd600c1
 #7 [ffff9ce03ecbbd70] __do_page_fault at ffffffff9fd746f0
 #8 [ffff9ce03ecbbde0] do_page_fault at ffffffff9fd74945
 #9 [ffff9ce03ecbbe10] page_fault at ffffffff9fd70778
    [exception RIP: memset+0x33]
    RIP: ffffffff9f9893f3  RSP: ffff9ce03ecbbec0  RFLAGS: 00010203
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: 00000000000003fe
    RDX: 000000000000fffe  RSI: 0000000000000000  RDI: 0000000000000000
    RBP: ffff9ce03ecbbf08   R8: ffffffffa0083fd1   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000000  R12: ffff9cd897eda000
    R13: ffff9ce05771e180  R14: ffff9cd897edf000  R15: 00007f47880021c0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff9ce03ecbbec0] get_current_proc_path.constprop.10 at ffffffffc06e3430 [probe_syscalls]
#11 [ffff9ce03ecbbf10] fh_sys_rename at ffffffffc06e473d [probe_syscalls]
#12 [ffff9ce03ecbbf50] system_call_fastpath at ffffffff9fd79ddb

  2. The logs can show page allocation failures:

    crash> log | grep 'page allocation failure'
[1111318.688145] virt-what: page allocation failure: order:5, mode:0x40d0
[1211927.164479] systemd-journal: page allocation failure: order:4, mode:0x40d0
[1609458.790525] bash: page allocation failure: order:5, mode:0x40d0
[1609458.872140] bashcf: page allocation failure: order:5, mode:0x40d0
[1609458.906452] 194-thirdparty-: page allocation failure: order:5, mode:0x40d0
[1655234.762811] user-provision: page allocation failure: order:5, mode:0x40d0
[1655235.184491] user-provision: page allocation failure: order:5, mode:0x40d0
[1655241.034045] net_pcap: page allocation failure: order:5, mode:0x40d0
[1729368.630354] 196-thirdparty-: page allocation failure: order:5, mode:0x40d0
[1753722.710457] systemd: page allocation failure: order:4, mode:0x40d0

  3. There are no higher order pages available:

    crash> pd ((struct zone *)0xffff9ce07ffda000)->free_area | grep nr_free | pr -Tn -N 0
    0       nr_free = 22550
    1       nr_free = 243
    2       nr_free = 0
    3       nr_free = 0
    4       nr_free = 0
    5       nr_free = 0
    6       nr_free = 0
    7       nr_free = 0
    8       nr_free = 0
    9       nr_free = 0
   10       nr_free = 0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments