Kernel Panic at audit_copy_inode() with NULL pointer dereference

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • sfc/onload modules
NAME            TAINTS
sfc_driverlink  OE
sfc_resource    OE
sfc             OE
sfc_char        OE
onload          OE

Issue

  • Server reboot with kernel panic message BUG: unable to handle kernel NULL pointer dereference at 0000000000000040).
  • RIP in function audit_copy_inode.

Resolution

  • Unfortunately, there is no resolution at this time, it is recommended to collect another vmcore by enabling slub_debug(slub_debug=FZP) which may be helpful in finding out the cause of corruption.
  • Based on our review of all support cases, wherever this issue occurs, 3rd-party sfc/onload modules are always installed and loaded.
  • Please check if the same issue persists even after blacklisting sfc/onload modules.

Root Cause

  • While constructing an audit record there is an invalid address that is NULL passed to inode structure inside audit_copy_inode() function causing the kernel to panic.

Diagnostic Steps

crash> sys | grep PANIC 
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000040"

crash> sys -i|head -n 5
        DMI_BIOS_VENDOR: Dell Inc.
       DMI_BIOS_VERSION: 2.11.2
          DMI_BIOS_DATE: 004/21/2021
         DMI_SYS_VENDOR: Dell Inc.
       DMI_PRODUCT_NAME: PowerEdge R740
  • Backtrace of panic task
crash> bt
PID: 116573   TASK: ffff91097b9f9080  CPU: 12   COMMAND: "publisherproc.e"
 #0 [ffff90ac2bd278b0] machine_kexec at ffffffffbda69514
 #1 [ffff90ac2bd27910] __crash_kexec at ffffffffbdb29e82
 #2 [ffff90ac2bd279e0] crash_kexec at ffffffffbdb29f78
 #3 [ffff90ac2bd279f8] oops_end at ffffffffbe1bc818
 #4 [ffff90ac2bd27a20] no_context at ffffffffbda7974c
 #5 [ffff90ac2bd27a70] __bad_area_nosemaphore at ffffffffbda79a2a
 #6 [ffff90ac2bd27ac0] bad_area_nosemaphore at ffffffffbda79b54
 #7 [ffff90ac2bd27ad0] __do_page_fault at ffffffffbe1bf8d0
 #8 [ffff90ac2bd27b40] do_page_fault at ffffffffbe1bfb05
 #9 [ffff90ac2bd27b70] page_fault at ffffffffbe1bb7b8
    [exception RIP: audit_copy_inode+0x2e]
    RIP: ffffffffbdb3fd1e  RSP: ffff90ac2bd27c20  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff908a3ec59860  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff908a3ec5989c  RDI: ffff908a3ec59860
    RBP: ffff90ac2bd27c58   R8: 0000000000000001   R9: ffffffffbeb3bbc0
    R10: 0000000000000000  R11: 0000000000000400  R12: 0000000000000000
    R13: ffff907461e4c300  R14: ffff908a3ec59860  R15: ffff904cb736fec0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff90ac2bd27c60] __audit_inode at ffffffffbdb46c97
#11 [ffff90ac2bd27cc0] do_last at ffffffffbdc6d3db
#12 [ffff90ac2bd27d70] path_openat at ffffffffbdc6daad
#13 [ffff90ac2bd27e08] do_filp_open at ffffffffbdc6f46d
#14 [ffff90ac2bd27ee0] do_sys_open at ffffffffbdc5ab34
#15 [ffff90ac2bd27f40] sys_open at ffffffffbdc5ac4e
#16 [ffff90ac2bd27f50] tracesys at ffffffffbe1c562e (via system_call)
    RIP: 00007f3dd3f7cee0  RSP: 00007ffc486f0f28  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 00007f3dd3f7f1b8  RCX: ffffffffffffffff
    RDX: 00007f3dd3cf6e82  RSI: 0000000000020002  RDI: 00007ffc486f0f30
    RBP: 00007ffc486f1030   R8: 0000000000000001   R9: 00000000024b4e96
    R10: 0000000000000003  R11: 0000000000000246  R12: 00007f3dd60f2046
    R13: 00000000024b4e94  R14: 00000000024b4ea2  R15: 00000000025b64c8
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b

open()flags = 20002 -> %400002 =  __O_NOFOLLOW|O_RDWR

 #8 [ffff90ac2bd27b40] do_page_fault at ffffffffbe1bfb05
    ffff90ac2bd27b48: ffff90ac2bd27bb8 0000000000000001 
    ffff90ac2bd27b58: 0000000000000000 ffff907461e4c300 
    ffff90ac2bd27b68: ffff90ac2bd27c58 ffffffffbe1bb7b8 
 #9 [ffff90ac2bd27b70] page_fault at ffffffffbe1bb7b8
    [exception RIP: audit_copy_inode+0x2e]
    RIP: ffffffffbdb3fd1e  RSP: ffff90ac2bd27c20  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff908a3ec59860  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff908a3ec5989c  RDI: ffff908a3ec59860
    RBP: ffff90ac2bd27c58   R8: 0000000000000001   R9: ffffffffbeb3bbc0
    R10: 0000000000000000  R11: 0000000000000400  R12: 0000000000000000
    R13: ffff907461e4c300  R14: ffff908a3ec59860  R15: ffff904cb736fec0
          ^- dentry               ^-audit_names
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

crash> dentry.d_inode ffff907461e4c300
  d_inode = 0xffff90edc7168af8,
  • publisherproc.e task on CPU 12 was trying to open the file /dev/shm/sem.foprod_11Net33JrnlPubGLTriggerEvent
crash> files -d ffff907461e4c300
     DENTRY           INODE           SUPERBLK     TYPE PATH
ffff907461e4c300 ffff90edc7168af8 ffff91097ebb4000 REG  /dev/shm/sem.foprod_11Net33JrnlPubGLTriggerEvent



    ffff90ac2bd27b78: ffff904cb736fec0 ffff908a3ec59860 
    ffff90ac2bd27b88: ffff907461e4c300 0000000000000000 
    ffff90ac2bd27b98: ffff90ac2bd27c58 ffff908a3ec59860 
    ffff90ac2bd27ba8: 0000000000000400 0000000000000000 
    ffff90ac2bd27bb8: ffffffffbeb3bbc0 0000000000000001 
    ffff90ac2bd27bc8: 0000000000000000 0000000000000000 
    ffff90ac2bd27bd8: 0000000000000000 ffff908a3ec5989c 
    ffff90ac2bd27be8: ffff908a3ec59860 

                                       ffffffffffffffff 
    ffff90ac2bd27bf8: ffffffffbdb3fd1e 0000000000000010 
    ffff90ac2bd27c08: 0000000000010246 ffff90ac2bd27c20 
    ffff90ac2bd27c18: 0000000000000018 

crash> filename 0xffff90c7d0a9b000
struct filename {
  name = 0xffff90c7d0a9b020 "/dev/shm/sem.foprod_11Net33JrnlPubGLTriggerEvent",
  uptr = 0x7ffc486f0f30 <error: Cannot access memory at address 0x7ffc486f0f30>,
  aname = 0xffff908a3ec59860,
  separate = 0x0,
  refcnt = 0x2
}


tos
                                       ffff90bb2f29d280 
    ffff90ac2bd27c28: ffff90bb2f29d280 ffff91097f3a85a8 
                                       rbx=inode
    ffff90ac2bd27c38: 000000003a02f4b2 0000000000000000 
                      r12              r13
    ffff90ac2bd27c48: ffff908a3ec59800 0000000000000000 
                      rbp
    ffff90ac2bd27c58: ffff90ac2bd27cb8 ffffffffbdb46c97 
#10 [ffff90ac2bd27c60] __audit_inode at ffffffffbdb46c97

  3114 static int do_last(struct nameidata *nd, struct path *path,
   3115                    struct file *file, const struct open_flags *op,
   3116                    int *opened, struct filename *name)
   3117 {
   3118         struct dentry *dir = nd->path.dentry;
   3119         int open_flag = op->open_flag;
   3120         bool will_truncate = (open_flag & O_TRUNC) != 0;
. . .

   3226          * create/update audit record if it already exists.
   3227          */
   3228         if (d_is_positive(path->dentry))
   3229               audit_inode(name, path->dentry, 0);  <---

   265 static inline void audit_inode(struct filename *name,
    266                                 const struct dentry *dentry,
    267                                 unsigned int flags) {
    268         if (unlikely(!audit_dummy_context())) {
    269                 unsigned int aflags = 0;
    270 
    271                 if (flags & LOOKUP_PARENT)
    272                         aflags |= AUDIT_INODE_PARENT;
    273                 if (flags & LOOKUP_NO_EVAL)
    274                         aflags |= AUDIT_INODE_NOEVAL;
    275                 __audit_inode(name, dentry, aflags);  <-----

   1783 void __audit_inode(struct filename *name, const struct dentry *dentry,
   1784                    unsigned int flags)
   1785 {
   1786         struct audit_context *context = current->audit_context;
   1787         struct inode *inode = dentry→d_inode;  <---
. .  .
  1871 out:    <---- jmp to here from  1830
   1872         if (parent) {
   1873                 n->name_len = n->name ? parent_len(n->name->name) : AUDIT_NAME_FULL;
   1874                 n->type = AUDIT_TYPE_PARENT;
   1875                 if (flags & AUDIT_INODE_HIDDEN)
   1876                         n->hidden = true;
   1877         } else {
   1878                 n->name_len = AUDIT_NAME_FULL;  <- -1
   1879                 n->type = AUDIT_TYPE_NORMAL;   <- 1
   1880         }
   1881         handle_path(dentry);

   1882         audit_copy_inode(n, dentry, inode, flags & AUDIT_INODE_NOEVAL);  ←--

  • While opening /dev/shm/sem.foprod_11Net33JrnlPubGLTriggerEvent, publisherproc.e was trying to construct an audit record for the inode but found that a null was passed in for the inode address.

  • When referencing inode->i_ino it caused a page fault that crashed the OS.

   1656 /* Copy inode data into an audit_names. */
   1657 void audit_copy_inode(struct audit_names *name, const struct dentry *dentry,
   1658                       struct inode *inode, unsigned int flags)
   1659 {

   1660         name->ino   = inode→i_ino;  <---
  • The inode address came from the dentry ffff907461e4c300.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments