Kernel Panic : "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008" with RIP on path_init

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Veritas vxfs (third party kernel module)

Issue

  • Server Panicked with vxfs messages on Red Hat Enterprise Linux

    vxfs: msgcnt 142 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/dg01/vol_ud file system full (8 block extent)
    
  • RIP in path_init+0x37f/0x3e0 called from third party module vxfs

Resolution

  • An exception occurred in unsigned vxfs kernel module. Contact the module vendor for further investigation.

Root Cause

  • The issue is because of dereferencing of an invalid address in RIP path_init by third-party module vxfs.

Diagnostic Steps

  • System Information

    crash> sys | grep  -e RELEASE -e PANIC
    RELEASE: 3.10.0-327.28.3.el7.x86_64
    PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008"
    
    
  • Kernel Ring Buffer:-

    [1665469.831436] vxfs: msgcnt 142 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/dg01/vol_ud file system full (8 block extent)
    [1665475.713496] vxfs: msgcnt 143 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/dg01/vol_ud file system full (8 block extent)
    [1665476.883108] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008       <<<<<<<<
    [1665476.883778] IP: [<ffffffff811eadaf>] path_init+0x37f/0x3e0      <<<<<<<<
    [1665476.883802] PGD aac6f067 PUD 7d97b067 PMD 0 
    [1665476.883820] Oops: 0000 [#1] SMP 
    [1665476.883833] Modules linked in: iptable_filter vxfen(POE) vxodm(POE) vxgms(POE) vxglm(POE) gab(POE) xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_
    srp scsi_transport_srp scsi_tgt llt(POE) ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr amf(POE) sisfim(POE) dmpaa(POE) vmw_vsock_vmci_transport vsock vxspec(POE) vxi
    o(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxdmp(POE) vxfs(POE) veki(POE) crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd sg pcspkr parport_pc parport vm
    w_vmci shpchp i2c_piix4 sisips(POE) binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel
    [1665476.884119]  vmwgfx serio_raw drm_kms_helper ttm mptsas vmw_pvscsi scsi_transport_sas drm mptscsih vmxnet3 mptbase ata_piix i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod
    [1665476.886969] CPU: 1 PID: 9968 Comm: uxioserv Tainted: P           OE  ------------   3.10.0-327.28.3.el7.x86_64 #1
    [1665476.888213] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
    [1665476.889462] task: ffff880006cd5c00 ti: ffff880387470000 task.ti: ffff880387470000
    [1665476.890678] RIP: 0010:[<ffffffff811eadaf>]  [<ffffffff811eadaf>] path_init+0x37f/0x3e0
    [1665476.891891] RSP: 0018:ffff8803874731d0  EFLAGS: 00010246
    [1665476.893075] RAX: ffff880006cd5c00 RBX: ffff8803874732f0 RCX: 0000000000000000
    [1665476.894240] RDX: 000000000000a27e RSI: ffff880035fa2020 RDI: ffff88043fc91d50
    [1665476.895382] RBP: ffff880387473210 R08: ffff880387473240 R09: 006578652f383639
    [1665476.896489] R10: 0000000000000000 R11: ffff88038747337e R12: 0000000000000040
    [1665476.897563] R13: ffff880035fa2020 R14: 00000000ffffff9c R15: 00000000000050c3
    [1665476.898611] FS:  00007f83529a1740(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
    [1665476.899648] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    [1665476.900680] CR2: 0000000000000008 CR3: 000000002272a000 CR4: 00000000000007e0
    [1665476.901747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [1665476.902802] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [1665476.903827] Stack:
    [1665476.904820]  ffffffffa04d4cec ffff880387473338 00000000b5862711 ffff880035fa2020
    [1665476.905822]  0000000000000040 ffff8803874732f0 00000000ffffff9c 00000000000050c3
    [1665476.906813]  ffff8803874732a8 ffffffff811ede05 0000000000000000 ffff880387473320
    [1665476.907776] Call Trace:
    [1665476.908771]  [<ffffffffa04d4cec>] ? vx_pri_getdele+0x10c/0xa90 [vxfs]      <<<<<<<<
    [1665476.909748]  [<ffffffff811ede05>] path_lookupat+0x45/0x7a0
    [1665476.910712]  [<ffffffff811ee58b>] filename_lookup+0x2b/0xc0
    [1665476.911659]  [<ffffffff811f0115>] kern_path+0x55/0xb0
    [1665476.912584]  [<ffffffff812fca02>] ? put_dec+0x72/0x90
    [1665476.913497]  [<ffffffff812fd9f3>] ? number.isra.2+0x323/0x360
    [1665476.914409]  [<ffffffffa030e1be>] sisips_path_lookup+0x2e/0x70 [sisips]
    [1665476.915349]  [<ffffffffa030e248>] ReadSymLink+0x48/0x100 [sisips]
    [1665476.916267]  [<ffffffff81300049>] ? snprintf+0x49/0x70
    [1665476.917197]  [<ffffffffa031b1c7>] _Z26GetProcessPathFromProcLinkiPci+0x47/0xe0 [sisips]
    [1665476.918124]  [<ffffffffa031b390>] ? _Z14SetProcessPathP7Process+0x30/0xa0 [sisips]
    [1665476.919363]  [<ffffffffa031ba70>] ? _Z12SetUpProcessP7Process+0xb0/0x180 [sisips]
    [1665476.920281]  [<ffffffffa030c6bd>] ? KeMutexEnter+0xd/0x10 [sisips]
    [1665476.921189]  [<ffffffffa0331aa6>] ? _ZN13ProcessCommon10GetProcessEi+0x96/0x100 [sisips]
    [1665476.922091]  [<ffffffffa0319c06>] ? AppfireCheckConnect+0x36/0x1e0 [sisips]
    [1665476.922985]  [<ffffffffa0309d5e>] ? hook_dgram_sendmsg+0xd2/0x113 [sisips]
    [1665476.923873]  [<ffffffffa0e2fd9a>] ? llt_udp_send+0x11a/0x380 [llt]
    [1665476.923873]  [<ffffffffa0e2fd9a>] ? llt_udp_send+0x11a/0x380 [llt]
    [1665476.924746]  [<ffffffff810d89ce>] ? getnstimeofday64+0xe/0x30
    [1665476.925620]  [<ffffffff810d8a19>] ? do_gettimeofday+0x29/0x70
    [1665476.926490]  [<ffffffffa0e3f7f3>] ? llt_transmit+0x323/0x580 [llt]
    [1665476.927358]  [<ffffffff81518fce>] ? __skb_clone+0x2e/0x130
    [1665476.928196]  [<ffffffffa0e42af2>] ? llt_send_port+0x222/0x410 [llt]
    [1665476.929008]  [<ffffffffa0e458a3>] ? llt_send_generic+0x663/0x800 [llt]
    [1665476.929805]  [<ffffffffa0e46263>] ? llt_send_data+0x253/0x300 [llt]
    [1665476.930561]  [<ffffffffa1038011>] ? gab_isend+0x271/0xc20 [gab]
    [1665476.931291]  [<ffffffffa1038be1>] ? gab_send_port_que+0x221/0xe30 [gab]
    [1665476.931992]  [<ffffffff81189b69>] ? zone_statistics+0x89/0xa0
    [1665476.932670]  [<ffffffffa1033240>] ? gab_qinsert+0x80/0x190 [gab]
    [1665476.933328]  [<ffffffffa1039a41>] ? gab_send_que+0x181/0x330 [gab]
    [1665476.933962]  [<ffffffffa102357a>] ? gab_send+0x11a/0x3a0 [gab]
    [1665476.934586]  [<ffffffffa10608d2>] ? vxg_gab_send_direct+0x72/0x80 [vxglm]
    [1665476.935217]  [<ffffffffa106097e>] ? vxg_send_one+0x8e/0xf0 [vxglm]
    [1665476.935818]  [<ffffffffa1060a6e>] ? vxg_send_msg+0x6e/0x90 [vxglm]
    [1665476.936423]  [<ffffffffa1072d95>] ? vxg_range_send_pbdata+0xb5/0xe0 [vxglm]
    [1665476.937020]  [<ffffffffa1072e04>] ? vxg_range_send_lock+0x44/0x60 [vxglm]
    [1665476.937613]  [<ffffffffa1072e9d>] ? vxg_range_start_msgwait_req+0x7d/0x90 [vxglm]
    [1665476.938209]  [<ffffffffa1072f8e>] ? vxg_range_msgwait_enter+0x6e/0xa0 [vxglm]
    [1665476.938793]  [<ffffffffa106e280>] ? vxg_range_cmn_lock+0xc0/0x2c0 [vxglm]
    [1665476.939376]  [<ffffffffa106d4f5>] ? vxg_lock_ilock_omnibus+0x1f5/0x280 [vxglm]
    [1665476.939952]  [<ffffffffa107006b>] ? vxg_api_range_lockwf+0x7b/0xa0 [vxglm]
    [1665476.940526]  [<ffffffffa106d580>] ? vxg_lock_ilock_omnibus+0x280/0x280 [vxglm]
    [1665476.941173]  [<ffffffffa0546ae6>] ? vx_glm_range_lock+0x46/0x50 [vxfs]
    [1665476.941912]  [<ffffffffa05412fc>] ? vx_glmrange_rangelock+0x7c/0xa0 [vxfs]
    [1665476.942522]  [<ffffffffa0498614>] ? vx_ipglock+0x34/0x40 [vxfs]
    [1665476.943134]  [<ffffffffa056e073>] ? vx_async_shorten+0x303/0x340 [vxfs]
    [1665476.943840]  [<ffffffffa057695b>] ? vx_irwlock+0x4b/0x70 [vxfs]
    [1665476.944429]  [<ffffffffa0605ded>] ? vx_do_frelease+0x2cd/0xa40 [vxfs]
    [1665476.945010]  [<ffffffffa06065a3>] ? vx_frelease+0x43/0xd0 [vxfs]
    [1665476.945562]  [<ffffffff811e0949>] ? __fput+0xe9/0x270
    [1665476.946111]  [<ffffffff811e0c0e>] ? ____fput+0xe/0x10
    [1665476.946657]  [<ffffffff810a2334>] ? task_work_run+0xc4/0xe0
    [1665476.947253]  [<ffffffff8108162b>] ? do_exit+0x2cb/0xa60
    [1665476.947810]  [<ffffffffa03316f0>] ? _ZN13ProcessCommon21CreateMissingChildrenEP7ProcessP15LIST_ENTRY_LINK+0x180/0x1f0 [sisips]
    [1665476.948390]  [<ffffffffa0330812>] ? _ZN8HashList14unlockHashLineEPv+0x12/0x50 [sisips]
    [1665476.948967]  [<ffffffffa030c6d2>] ? KeMutexExit+0x12/0x20 [sisips]
    [1665476.949552]  [<ffffffffa032ea96>] ? _ZN7Process15releaseInternalEv+0x46/0xf0 [sisips]
    [1665476.950142]  [<ffffffff81081e3f>] ? do_group_exit+0x3f/0xa0
    [1665476.950713]  [<ffffffff81081eb4>] ? SyS_exit_group+0x14/0x20
    [1665476.951284]  [<ffffffffa030739f>] ? hook_exit_group+0x6e/0x8e [sisips]
    [1665476.951852]  [<ffffffff81646b49>] ? system_call_fastpath+0x16/0x1b
    
    
  • Backtraces of the panic task

    crash> bt
    PID: 9968   TASK: ffff880006cd5c00  CPU: 1   COMMAND: "uxioserv"
    #0 [ffff880387472e98] machine_kexec at ffffffff81051e9b
    #1 [ffff880387472ef8] crash_kexec at ffffffff810f27a2
    #2 [ffff880387472fc8] oops_end at ffffffff8163f448
    #3 [ffff880387472ff0] no_context at ffffffff8162f57b
    #4 [ffff880387473040] __bad_area_nosemaphore at ffffffff8162f611
    #5 [ffff880387473088] bad_area_nosemaphore at ffffffff8162f77b
    #6 [ffff880387473098] __do_page_fault at ffffffff816421be
    #7 [ffff8803874730f8] do_page_fault at ffffffff81642353
    #8 [ffff880387473120] page_fault at ffffffff8163e648
       [exception RIP: path_init+895]
       RIP: ffffffff811eadaf  RSP: ffff8803874731d0  RFLAGS: 00010246
       RAX: ffff880006cd5c00  RBX: ffff8803874732f0  RCX: 0000000000000000
       RDX: 000000000000a27e  RSI: ffff880035fa2020  RDI: ffff88043fc91d50
       RBP: ffff880387473210   R8: ffff880387473240   R9: 006578652f383639
       R10: 0000000000000000  R11: ffff88038747337e  R12: 0000000000000040
       R13: ffff880035fa2020  R14: 00000000ffffff9c  R15: 00000000000050c3
       ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    #9 [ffff8803874731d0] vx_pri_getdele at ffffffffa04d4cec [vxfs]    <<<<<<<<
    #10 [ffff8803874732b0] filename_lookup at ffffffff811ee58b
    #11 [ffff8803874732e8] kern_path at ffffffff811f0115
    #12 [ffff8803874733b8] sisips_path_lookup at ffffffffa030e1be [sisips]
    #13 [ffff8803874733f8] ReadSymLink at ffffffffa030e248 [sisips]
    #14 [ffff8803874734c8] _Z26GetProcessPathFromProcLinkiPci at ffffffffa031b1c7 [sisips]
    #15 [ffff8803874734e8] _Z14SetProcessPathP7Process at ffffffffa031b390 [sisips]
    #16 [ffff880387473508] _Z12SetUpProcessP7Process at ffffffffa031ba70 [sisips]
    #17 [ffff880387473538] _ZN13ProcessCommon10GetProcessEi at ffffffffa0331aa6 [sisips]
    #18 [ffff880387473558] AppfireCheckConnect at ffffffffa0319c06 [sisips]
    #19 [ffff8803874735e0] hook_dgram_sendmsg at ffffffffa0309d5e [sisips]
    #20 [ffff880387473648] llt_udp_send at ffffffffa0e2fd9a [llt]
    #21 [ffff880387473718] llt_transmit at ffffffffa0e3f7f3 [llt]
    #22 [ffff880387473760] llt_send_port at ffffffffa0e42af2 [llt]
    #23 [ffff8803874737d0] llt_send_generic at ffffffffa0e458a3 [llt]
    #24 [ffff880387473870] llt_send_data at ffffffffa0e46263 [llt]
    #25 [ffff8803874738c0] gab_isend at ffffffffa1038011 [gab]
    #26 [ffff880387473968] gab_send_port_que at ffffffffa1038be1 [gab]
    #27 [ffff880387473a20] gab_send_que at ffffffffa1039a41 [gab]
    #28 [ffff880387473a80] gab_send at ffffffffa102357a [gab]
    #29 [ffff880387473ad0] vxg_gab_send_direct at ffffffffa10608d2 [vxglm]
    #30 [ffff880387473af0] vxg_send_one at ffffffffa106097e [vxglm]
    #31 [ffff880387473b20] vxg_send_msg at ffffffffa1060a6e [vxglm]
    #32 [ffff880387473b48] vxg_range_send_pbdata at ffffffffa1072d95 [vxglm]
    #33 [ffff880387473b90] vxg_range_send_lock at ffffffffa1072e04 [vxglm]
    #34 [ffff880387473bb8] vxg_range_start_msgwait_req at ffffffffa1072e9d [vxglm]
    #35 [ffff880387473be0] vxg_range_msgwait_enter at ffffffffa1072f8e [vxglm]
    #36 [ffff880387473c08] vxg_range_cmn_lock at ffffffffa106e280 [vxglm]
    #37 [ffff880387473c58] vxg_api_range_lockwf at ffffffffa107006b [vxglm]
    #38 [ffff880387473cc8] vx_glm_range_lock at ffffffffa0546ae6 [vxfs]
    #39 [ffff880387473cd8] vx_glmrange_rangelock at ffffffffa05412fc [vxfs]
    #40 [ffff880387473ce8] vx_ipglock at ffffffffa0498614 [vxfs]
    #41 [ffff880387473cf8] vx_async_shorten at ffffffffa056e073 [vxfs]
    #42 [ffff880387473d58] vx_do_frelease at ffffffffa0605ded [vxfs]
    #43 [ffff880387473dd0] vx_frelease at ffffffffa06065a3 [vxfs]
    #44 [ffff880387473e00] __fput at ffffffff811e0949
    #45 [ffff880387473e48] ____fput at ffffffff811e0c0e
    #46 [ffff880387473e58] task_work_run at ffffffff810a2334
    #47 [ffff880387473e88] do_exit at ffffffff8108162b
    #48 [ffff880387473f18] do_group_exit at ffffffff81081e3f
    #49 [ffff880387473f48] sys_exit_group at ffffffff81081eb4
    #50 [ffff880387473f58] hook_exit_group at ffffffffa030739f [sisips]
    #51 [ffff880387473f80] system_call_fastpath at ffffffff81646b49
       RIP: 00007f834a965899  RSP: 00007ffc277aec00  RFLAGS: 00010202
       RAX: 00000000000000e7  RBX: ffffffff81646b49  RCX: 0000000000000001
       RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000000
       RBP: 00007f834ac62838   R8: 000000000000003c   R9: 00000000000000e7
       R10: ffffffffffffff70  R11: 0000000000000246  R12: 0000000000000000
       R13: 0000000000000000  R14: 000000008110b9d4  R15: ffffffffa030739f
    ORIG_RAX: 00000000000000e7  CS: 0033  SS: 002b
    
    
  • Disassembly of RIP ffffffff811eadaf

    crash> dis -lr ffffffff811eadaf|tail
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/include/linux/seqlock.h: 78
    0xffffffff811ead9f <path_init+879>:  test   $0x1,%al
    0xffffffff811eada1 <path_init+881>:  jne    0xffffffff811eadf9 <path_init+969>
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/namei.c: 714
    0xffffffff811eada3 <path_init+883>:  mov    %eax,0x3c(%rbx)
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/namei.c: 715
    0xffffffff811eada6 <path_init+886>:  cmp    0x8(%rcx),%esi
    0xffffffff811eada9 <path_init+889>:  je     0xffffffff811eac29 <path_init+505>
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/include/linux/seqlock.h: 77
    0xffffffff811eadaf <path_init+895>:  mov    0x8(%rcx),%esi   <<<<<<<< [kernel panicked here]
    
    
  • The corresponding kernel source code: fs/namei.c are given below.

    705 static __always_inline void set_root_rcu(struct nameidata *nd)
    706 {
    707         if (!nd->root.mnt) {
    708                 struct fs_struct *fs = current->fs;
    709                 unsigned seq;
    710 
    711                 do {
    712                         seq = read_seqcount_begin(&fs->seq);
    713                         nd->root = fs->root;
    714                         nd->seq = __read_seqcount_begin(&nd->root.dentry->d_seq);      <<<<<<<
    
    
    72 static inline unsigned __read_seqcount_begin(const seqcount_t *s)
    73 {
    74         unsigned ret;
    75 
    76 repeat:
    77         ret = ACCESS_ONCE(s->sequence);      <<<<<<<kernel panicked here
    
    
  • The content of the function path_init

    crash> px path_init
    path_init = $1 = 
    {int (int, const char *, unsigned int, struct nameidata *, struct file **)} 0xffffffff811eaa30 <path_init>
    
  • Disassembly of address ffffffffa04d4cec to check the value of %rcx.

    crash> dis -r ffffffffa04d4cec|tail -5
    0xffffffffa04d4cdb <vx_pri_getdele+251>: mov    -0x44(%rbp),%edx
    0xffffffffa04d4cde <vx_pri_getdele+254>: mov    %r13,0x8(%rsp)
    0xffffffffa04d4ce3 <vx_pri_getdele+259>: mov    %rax,(%rsp)
    0xffffffffa04d4ce7 <vx_pri_getdele+263>: callq  0xffffffffa04d48c0 <vx_getedele_size>
    0xffffffffa04d4cec <vx_pri_getdele+268>: mov    -0x30(%rbp),%rcx    <<<<<<< %rcx got the value from here
    
    
  • Backtraces for the function vx_pri_getdele

    crash> bt -f |grep -B 10 "vx_pri_getdele"
       ffff880387473138: ffff880035fa2020 0000000000000040 
       ffff880387473148: ffff880387473210 ffff8803874732f0 
       ffff880387473158: ffff88038747337e 0000000000000000 
       ffff880387473168: 006578652f383639 ffff880387473240 
       ffff880387473178: ffff880006cd5c00 0000000000000000 
       ffff880387473188: 000000000000a27e ffff880035fa2020 
       ffff880387473198: ffff88043fc91d50 ffffffffffffffff   <<<<<<< 0x30(%rbp)
       ffff8803874731a8: ffffffff811eadaf 0000000000000010 
       ffff8803874731b8: 0000000000010246 ffff8803874731d0 
       ffff8803874731c8: 0000000000000018 ffffffffa04d4cec 
    #9 [ffff8803874731d0] vx_pri_getdele at ffffffffa04d4cec [vxfs]
    
    
  • From the source code, below line was observed earlier.

    705 static __always_inline void set_root_rcu(struct nameidata *nd)   <<<<<<<
    706 {
    
  • The value in *nd = ffff88043fc91d50

  • Now the value in nameidata is found as below.

    crash> struct nameidata.root ffff88043fc91d50 -ox
    struct nameidata {
     [ffff88043fc91d70] struct path root;
    
    }
    crash> struct path ffff88043fc91d70 -ox
    struct path {
     [ffff88043fc91d70] struct vfsmount *mnt;
     [ffff88043fc91d78] struct dentry *dentry;     <<<<<<< 
    
    
  • The below information looks invalid.

    crash> struct vfsmount ffff88043fc91d70
    struct vfsmount {
     mnt_root = 0x0, 
     mnt_sb = 0x0, 
     mnt_flags = 0
    }
    
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/arch/x86/include/asm/current.h: 14
    0xffffffff811ead6f <path_init+831>:     mov    %gs:0xb7c0,%rax
    
    crash> kmem -o |grep "CPU 1:"
     CPU 1: ffff88043fc80000
    
    crash> px 0xffff88043fc80000+0xb7c0
    $8 = 0xffff88043fc8b7c0
    
    crash> rd 0xffff88043fc8b7c0
    ffff88043fc8b7c0:  ffff880006cd5c00
    
    /usr/src/debug/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/namei.c: 708
    0xffffffff811ead78 <path_init+840>:     mov    0x750(%rax),%rcx
    0xffffffff811ead7f <path_init+847>:     jmp    0xffffffff811eadaf <path_init+895>
    0xffffffff811ead81 <path_init+849>:     nopl   0x0(%rax)
    
       705 static __always_inline void set_root_rcu(struct nameidata *nd)
       706 {
       707         if (!nd->root.mnt) {
       708                 struct fs_struct *fs = current->fs;    <<<<< fs_struct coming from current->fs
    
    crash> struct task_struct -ox|grep fs_struct
     [0x750] struct fs_struct *fs;
    
    crash> struct task_struct.fs ffff880006cd5c00
     fs = 0x0       <<<<<<<< %rcx
    
    
  • The value of %rcx is 0000000000000000.

    crash> task -R fs
    PID: 9968   TASK: ffff880006cd5c00  CPU: 1   COMMAND: "uxioserv"
     fs = 0x0, 
    
  • The instruction which caused panic "mov 0x8(%rcx),%esi"

    crash> struct fs_struct -ox |grep seq
      [0x8] seqcount_t seq;
    crash> px 0x0000000000000000+0x8
    $4 = 0x8
    
  • This is the vaule in the panic message.

    crash> sys|grep PANIC
          PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000008" 
    
  • Checking the current->fs which passed NULL to fs_structure.
  • First, checking filesystem information from __fput () function when the process started.


    crash> px __fput __fput = $9 = {void (struct file *)} 0xffffffff811e0860 <__fput> crash> dis -r ffffffff810a2334|tail -3 0xffffffff810a232e <task_work_run+190>: mov %rdx,%rdi <<<<<<<< 0xffffffff810a2331 <task_work_run+193>: callq *0x8(%rdx) 0xffffffff810a2334 <task_work_run+196>: callq 0xffffffff8163bdc0 <_cond_resched> ..... 0xffffffff811e0872 <__fput+18>: mov %rdi,%rbx <<<<<<<< ..... 0xffffffff811e0949 <__fput+233>: mov %rbx,%rdi <<<<<<<< crash> bt -f|grep -A 4 vx_frelease #43 [ffff880387473dd0] vx_frelease at ffffffffa06065a3 [vxfs] ffff880387473dd8: ffff880352a0fe00 0000000000000008 %rbx ffff880387473de8: ffff880047039d50 ffff8801254f90c0 ffff880387473df8: ffff880387473e40 ffffffff811e0949 #44 [ffff880387473e00] __fput at ffffffff811e0949
  • Value of %rdi = ffff880352a0fe00

    crash> struct file.f_path ffff880352a0fe00
     f_path = {
       mnt = 0xffff880404948720, 
       dentry = 0xffff8801254f90c0
     }
    
  • The below information looks valid.

    crash> struct vfsmount 0xffff880404948720
    struct vfsmount {
     mnt_root = 0xffff8803d9cf3e40, 
     mnt_sb = 0xffff88040481a800, 
     mnt_flags = 4096
    }
    
  • The task uxioserv was accessing vxfs filesystem.

    crash> struct super_block.s_type 0xffff88040481a800
     s_type = 0xffffffffa06fb820
    
    crash> sym 0xffffffffa06fb820
    ffffffffa06fb820 (d) vx_fs_type [vxfs]
    
    
  • The task uxioserv was accessing universe.log file which was mounted /ud.

    crash> files -d 0xffff8801254f90c0
        DENTRY           INODE           SUPERBLK     TYPE PATH
    ffff8801254f90c0 ffff880047039d50 ffff88040481a800 REG  /ud/apps/path/to/log/universe.log
    
    crash> mount |grep "ffff88040481a800\|NAME"
        MOUNT           SUPERBLK     TYPE   DEVNAME   DIRNAME
    ffff880404948700 ffff88040481a800 vxfs   /dev/vx/dsk/dg01/vol_ud /ud 
    
    
  • verifying the filesystem information from kern_path() function

    crash> px kern_path
    kern_path = $10 = 
    {int (const char *, unsigned int, struct path *)} 0xffffffff811f00c0 
    <kern_path>
    
    crash> dis -r ffffffffa030e1be|tail -6
    0xffffffffa030e1ad <sisips_path_lookup+29>:  mov    %rbx,%rdi  
    0xffffffffa030e1b0 <sisips_path_lookup+32>:  mov    %rsp,%rdx  <<<<<<<
    0xffffffffa030e1b3 <sisips_path_lookup+35>:  mov    %r14d,%esi
    0xffffffffa030e1b6 <sisips_path_lookup+38>:  mov    %eax,%r12d
    0xffffffffa030e1b9 <sisips_path_lookup+41>:  callq  0xffffffff811f00c0 <kern_path>
    0xffffffffa030e1be <sisips_path_lookup+46>:  test   %eax,%eax
    ....
    0xffffffff811f00c6 <kern_path+6>:       mov    %rsp,%rbp
    ....
    
    crash> dis -r ffffffff811ee58b|head -3
    0xffffffff811ee560 <filename_lookup>:    data32 data32 data32 xchg %ax,%ax [FTRACE NOP]
    0xffffffff811ee565 <filename_lookup+5>:  push   %rbp   <<<<<<< %rdx
    0xffffffff811ee566 <filename_lookup+6>:  mov    %rsp,%rbp
    
    crash> bt -f |grep -A 5 filename_lookup
    #10 [ffff8803874732b0] filename_lookup at ffffffff811ee58b
       ffff8803874732b8: ffff8803874732e0 ffff880035fa2000 
       ffff8803874732c8: 0000000000000000 ffff8803874733c0 
       ffff8803874732d8: 0000000000000000 ffff8803874733b0 
    <<<<<<< %rdx
       ffff8803874732e8: ffffffff811f0115 
    #11 [ffff8803874732e8] kern_path at ffffffff811f0115
    
    crash> struct path ffff8803874733b0
    struct path {
     mnt = 0xffff8803874733f0, 
     dentry = 0xffffffffa030e1be  <<<<<<<<
    }
    
    
  • The above dentry pointer of vxfs file system was invalid, which is pointing to address of unsigned (U) sisips module.

    crash> sym 0xffffffffa030e1be
    ffffffffa030e1be (t) sisips_path_lookup+46 [sisips] 
    
  • The same information can be verified from filename_lookup() function.

    crash> px filename_lookup
    filename_lookup = $13 = 
    {int (int, struct filename *, unsigned int, struct nameidata *)} 0xffffffff811ee560 <filename_lookup>
    
    crash> struct nameidata ffff8803874732e0
    struct nameidata {
     path = {
       mnt = 0xffff8803874733b0, 
       dentry = 0xffffffff811f0115 <kern_path+85>
     }, 
     last = {
       {
         {
           hash = 3189, 
           len = 256
         }, 
         hash_len = 1099511630965
       }, 
       name = 0xff30880411e64c00 <Address 0xff30880411e64c00 out of bounds>
     }, 
     root = {
       mnt = 0x100000001, 
       dentry = 0x45d40f
     }, 
     inode = 0x0, 
     flags = 1, 
     seq = 0, 
     last_type = 0, 
     depth = 1, 
     saved_names = {0xffff880300001050 "", 0x1 <Address 0x1 out of bounds>, 0xffffffff812fca02 <put_dec+114> "[A\\]\303f\017\037\204", 0xffff880035fa0006 "9968/exe", 0xffff880035fa1000 "", 0xffff8803874733f0 "\300\064G\207\003\210\377\377H\342\060\240\377\377\377\377\377\377G\207\003\210\377\377\021'\206\265", 0xffffffff812fd9f3 <number+803> "L\213\235x\377\377\377D\213\215p\377\377\377D\213\205t\377\377\377D\213U\200L)؉ƍH\377\351\177\375\377\377H\211\330\351\240\376\377\377\350\312\325\327\377f.\017\037\204", 0xa0000ffff <Address 0xa0000ffff out of bounds>, 0xffff88038747337e "8699"}
    }
    
    
  • The same invalid dentry pointer was passed to super_block in filename_lookup() function.

    crash> struct vfsmount  0xffff8803874733b0
    struct vfsmount {
     mnt_root = 0xffff8803874733f0, 
     mnt_sb = 0xffffffffa030e1be,   <<<<<<<<
     mnt_flags = -1249499375
    }
    
    crash> kmem 0xffffffffa030e1be
    ffffffffa030e1be (t) sisips_path_lookup+46 [sisips] 
    
      VMAP_AREA         VM_STRUCT                 ADDRESS RANGE                SIZE
    ffff880423d60080  ffff8804250ff2c0  ffffffffa0307000 - ffffffffa0352000   307200
    
         PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
    ffffea001092b140 424ac5000                0        0  1 2fffff00000000
    
    
  • The unsigned (U) sisips module was loaded on the system.

    crash> mod -t |grep sisips
    sisips    POE
    
    crash> mod |grep "NAME\|sisips"
        MODULE       NAME                         SIZE  OBJECT FILE
    ffffffffa033b280  sisips                     300089  (not loaded)  [CONFIG_KALLSYMS]
    
    crash> module ffffffffa033b280|egrep version
     version = 0x0, 
     srcversion = 0x0, 
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.