Kernel panic at `lockref_put_or_lock()` due to an issue with unsigned module.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • 3rd party module sysevt
  • 3rd party module mvfs

Issue

  • System panic with the below log.
 BUG: unable to handle kernel NULL pointer dereference at 0000000000000056
 IP: [<ffffffff9695fa95>] lockref_put_or_lock+0x5/0x80
..
 CPU: 2 PID: 1714 Comm: nfsd Kdump: loaded Tainted: P           OE  ------------   3.10.0-862.11.6.el7.x86_64 #1
 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
 RIP: 0010:[<ffffffff9695fa95>]  [<ffffffff9695fa95>] lockref_put_or_lock+0x5/0x80
..
 Call Trace:
  [<ffffffff968376b9>] dput+0x29/0x160
  [<ffffffffc0512472>] sisevt_nfsd4_proc_compound+0x1b2/0xab0 [sisevt]
  [<ffffffffc079c590>] ? nfsd_dispatch+0xe0/0x290 [nfsd]
  [<ffffffffc0369866>] ? svc_process_common+0x466/0x710 [sunrpc]
  [<ffffffffc0369c13>] ? svc_process+0x103/0x190 [sunrpc]
  [<ffffffffc079beef>] ? nfsd+0xdf/0x150 [nfsd]
  [<ffffffffc079be10>] ? nfsd_destroy+0x80/0x80 [nfsd]
  [<ffffffff966bdf21>] ? kthread+0xd1/0xe0
  [<ffffffff966bde50>] ? insert_kthread_work+0x40/0x40
  [<ffffffff96d255f7>] ? ret_from_fork_nospec_begin+0x21/0x21
  [<ffffffff966bde50>] ? insert_kthread_work+0x40/0x40
 RIP  [<ffffffff9695fa95>] lockref_put_or_lock+0x5/0x80
  • In another case, kernel panic with below log.

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000056
 IP: [<ffffffffb7396ed5>] lockref_put_or_lock+0x5/0x80
..
 CPU: 3 PID: 7065 Comm: rg Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-1127.el7.x86_64 #1
 Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 05/21/2018
 RIP: 0010:[<ffffffffb7396ed5>]  [<ffffffffb7396ed5>] lockref_put_or_lock+0x5/0x80
..
 Call Trace:
  [<ffffffffb7266a49>] dput+0x29/0x1a0
  [<ffffffffc0919e6f>] vnlayer_linux_free_clrvnode+0x26/0x47 [mvfs]
  [<ffffffffc0918f14>] mvfs_clear_inode+0xe0/0xfb [mvfs]
  [<ffffffffc0918f41>] mvfs_evict_inode+0x12/0x2b [mvfs]
  [<ffffffffb726b674>] evict+0xb4/0x180
  [<ffffffffb726ba9c>] iput+0xfc/0x190
  [<ffffffffc091c45d>] mdki_vn_rele+0x12/0x14 [mvfs]
..
  [<ffffffffb725342e>] SyS_newstat+0xe/0x10
  [<ffffffffb7792ed2>] system_call_fastpath+0x25/0x2a

Resolution

When running with sisevt

  • Please contact the module sysev vendor for a possible resolution.

When running with mvfs

  • Please contact the module mvfs vendor for a possible resolution.

Diagnostic Steps

  • The system crashed due to invalid memory access.
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000056"
  • This was happening while trying to get lock for a dentry as this dentry had invalid address 0xfffffffffffffffe.
crash> bt
PID: 1714     TASK: ffff9a252eb50fd0  CPU: 2    COMMAND: "nfsd"
 #0 [ffff9a257fa5b9d8] machine_kexec at ffffffff966629da
 #1 [ffff9a257fa5ba38] __crash_kexec at ffffffff96716692
 #2 [ffff9a257fa5bb08] crash_kexec at ffffffff96716780
 #3 [ffff9a257fa5bb20] oops_end at ffffffff96d1d728
 #4 [ffff9a257fa5bb48] no_context at ffffffff96d0c6cd
 #5 [ffff9a257fa5bb98] __bad_area_nosemaphore at ffffffff96d0c764
 #6 [ffff9a257fa5bbe8] bad_area_nosemaphore at ffffffff96d0c8d5
 #7 [ffff9a257fa5bbf8] __do_page_fault at ffffffff96d206e0
 #8 [ffff9a257fa5bc60] do_page_fault at ffffffff96d208d5
 #9 [ffff9a257fa5bc90] page_fault at ffffffff96d1c758
    [exception RIP: lockref_put_or_lock+0x5]
    RIP: ffffffff9695fa95  RSP: ffff9a257fa5bd48  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: fffffffffffffffe  RCX: ffff9a257fa5bfd8
    RDX: 0000000000000000  RSI: ffffffffffffffff  RDI: 0000000000000056
    RBP: ffff9a257fa5bd50   R8: 0000000002000000   R9: 6138313030303030
    R10: 6565303061383130  R11: 0000000000000001  R12: 0000000000000056
    R13: ffff9a25b3ec0c00  R14: 0000000002000000  R15: 0000000000000001
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff9a257fa5bd58] dput at ffffffff968376b9
#11 [ffff9a257fa5bd88] sisevt_nfsd4_proc_compound at ffffffffc0512472 [sisevt]
#12 [ffff9a257fa5bde0] nfsd_dispatch at ffffffffc079c590 [nfsd]
#13 [ffff9a257fa5be18] svc_process_common at ffffffffc0369866 [sunrpc]
#14 [ffff9a257fa5be78] svc_process at ffffffffc0369c13 [sunrpc]
#15 [ffff9a257fa5bea0] nfsd at ffffffffc079beef [nfsd]
#16 [ffff9a257fa5bec8] kthread at ffffffff966bdf21

0xffffffff9695fa94 <lockref_put_or_lock+0x4>:   push   %rbx    ; 0xfffffffffffffffe  <-- Invalid dentry value

  • This invalid dentry address was generated in sisevt and doesn't show anywhere before that.

  • In another case, with mvfs module

crash> bt
PID: 7065     TASK: ffff9b0db02b1070  CPU: 3    COMMAND: "rg"
 #0 [ffff9b0b756ff760] machine_kexec at ffffffffb7066044
 #1 [ffff9b0b756ff7c0] __crash_kexec at ffffffffb7122ee2
 #2 [ffff9b0b756ff890] crash_kexec at ffffffffb7122fd0
 #3 [ffff9b0b756ff8a8] oops_end at ffffffffb778a798
 #4 [ffff9b0b756ff8d0] no_context at ffffffffb7075d74
 #5 [ffff9b0b756ff920] __bad_area_nosemaphore at ffffffffb7076042
 #6 [ffff9b0b756ff970] bad_area_nosemaphore at ffffffffb7076164
 #7 [ffff9b0b756ff980] __do_page_fault at ffffffffb778d750
 #8 [ffff9b0b756ff9f0] do_page_fault at ffffffffb778d975
 #9 [ffff9b0b756ffa20] page_fault at ffffffffb7789778
    [exception RIP: lockref_put_or_lock+5]
    RIP: ffffffffb7396ed5  RSP: ffff9b0b756ffad0  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: fffffffffffffffe  RCX: ffff9b0b756fffd8
    RDX: 0000000000000000  RSI: e000000000000000  RDI: 0000000000000056
    RBP: ffff9b0b756ffad8   R8: 8038000000000000   R9: 0dcd57fc401c0000
    R10: ffff9b01cf8df0a0  R11: ffffed29b13a4940  R12: ffff9b0dcd57fca8
    R13: 0000000000000056  R14: ffffffffc093b2c0  R15: ffff9b0069db2020
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff9b0b756ffae0] dput at ffffffffb7266a49
#11 [ffff9b0b756ffb10] vnlayer_linux_free_clrvnode at ffffffffc0919e6f [mvfs]
#12 [ffff9b0b756ffb28] mvfs_clear_inode at ffffffffc0918f14 [mvfs]
#13 [ffff9b0b756ffb58] mvfs_evict_inode at ffffffffc0918f41 [mvfs]
#14 [ffff9b0b756ffb70] evict at ffffffffb726b674
#15 [ffff9b0b756ffb98] iput at ffffffffb726ba9c
#16 [ffff9b0b756ffbc8] mdki_vn_rele at ffffffffc091c45d [mvfs]
#17 [ffff9b0b756ffbd8] vnlayer_dent_cvn_rele at ffffffffc09176f9 [mvfs]
#18 [ffff9b0b756ffc00] vnode_shadow_dop_release at ffffffffc0917764 [mvfs]
#19 [ffff9b0b756ffc18] vnode_dop_release at ffffffffc09113c4 [mvfs]
#20 [ffff9b0b756ffc28] d_free at ffffffffb7265abc
#21 [ffff9b0b756ffc48] __dentry_kill at ffffffffb7266440
#22 [ffff9b0b756ffc70] dput at ffffffffb7266ad5
#23 [ffff9b0b756ffca0] terminate_walk at ffffffffb7259181
#24 [ffff9b0b756ffcb8] path_lookupat at ffffffffb725beba
#25 [ffff9b0b756ffd50] filename_lookup at ffffffffb725c5cb
#26 [ffff9b0b756ffd88] user_path_at_empty at ffffffffb72602d7
#27 [ffff9b0b756ffe58] user_path_at at ffffffffb7260341
#28 [ffff9b0b756ffe68] vfs_fstatat at ffffffffb7252bb3
#29 [ffff9b0b756ffeb8] SYSC_newstat at ffffffffb7252f6e
#30 [ffff9b0b756fff40] sys_newstat at ffffffffb725342e
#31 [ffff9b0b756fff50] system_call_fastpath at ffffffffb7792ed2
    RIP: 00007fb19c9deb6f  RSP: 00007fb19a5fd6f8  RFLAGS: 00000246
    RAX: 0000000000000004  RBX: 00007fb19a5fcca0  RCX: ffffffffffffffff
    RDX: 00007fb19a5fcca0  RSI: 00007fb19a5fcbf0  RDI: 00007fb19a5fcd38
    RBP: 00007fb19a5fcbf0   R8: 00000000ffffff9c   R9: 00007fb19a5fcd38
    R10: fefefefefefefeff  R11: 0000000000000246  R12: 00007fb196e01600
    R13: 00007fb19a0291f0  R14: 00007fb19a5fcf60  R15: 00007fb19a0291f2
    ORIG_RAX: 0000000000000004  CS: 0033  SS: 002b

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments