Kernel panic during NULL pointer dereference at 0000000000000000 inside PowerPlatformBottomDispatch() function.

Diagnostic Steps

• Verify kernel ring buffer for following call traces.

Kernel Ring Buffer:

crash> log
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: 
 [<ffffffff8850ec09>] :emcp:PowerPlatformBottomDispatch+0x329/0x760
PGD f8e3c3067 PUD 809cd0067 PMD 0 
Oops: 0000 [1] SMP 
last sysfs file: /block/sdi/stat
CPU 19 
Modules linked in: emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpmpx(PU) emcpgpx(PU) emcp(PU) ipv6 xfrm_nalgo crypto_api nfs nfs_acl oracleasm(U) autofs4 adm1021 lockd sunrpc dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev fnic libfcoe i7core_edac libfc i2c_i801 enic pcspkr edac_mc sr_mod i2c_core cdrom scsi_transport_fc tpm_tis tpm tpm_bios sg 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage shpchp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 26035, comm: MpxPeriodicCall Tainted: P     ---- 2.6.18-308.8.1.el5 #1
RIP: 0010:[<ffffffff8850ec09>]  [<ffffffff8850ec09>] :emcp:PowerPlatformBottomDispatch+0x329/0x760
RSP: 0000:ffff814072c75a10  EFLAGS: 00010206
RAX: 00000000000001dc RBX: 0000000000000000 RCX: 00000000000001d0
RDX: ffff81307be7ded0 RSI: 0000000000000100 RDI: ffff81407d47d008
RBP: 00000000085001c0 R08: ffff81307fdaa608 R09: 0000000000000000
R10: ffff812b5e6f5880 R11: 00000000000000e0 R12: ffff8129b9d05788
R13: 00000000000001dc R14: ffff812f27b83008 R15: ffff812f27b83080
FS:  00002ba09472f6e0(0000) GS:ffff8120b83db340(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000ce85bc000 CR4: 00000000000006a0
Process MpxPeriodicCall (pid: 26035, threadinfo ffff814072c74000, task ffff81407b484100)
Stack:  ffff81407c88f2c0 0000000000000246 0000000000000010 ffff81407f8734c0
 ffff812b5e6f5880 ffff81407d47d008 0000000000000000 ffff81407d47d030
 ffff812b5e6f5880 ffff812f27b83008 ffff81406e40ec80 ffffffff8850f0d8
Call Trace:
 [<ffffffff8850f0d8>] :emcp:PowerSyncIoBottomDispatch+0x78/0xd0
 [<ffffffff8850f626>] :emcp:PowerDispatchX+0x356/0x410
 [<ffffffff885107fd>] :emcp:EmsInquiry+0x9d/0x1a0
 [<ffffffff8873e966>] :emcpmpx:SymmetrixKLam_getPathLunStatus+0x36/0xf0
 [<ffffffff80046bba>] try_to_wake_up+0x472/0x484
 [<ffffffff8872d391>] :emcpmpx:MpxDefaultTestPath+0x21/0x80
 [<ffffffff8873cbf3>] :emcpmpx:MpxLnxTestPath+0x43/0x270
 [<ffffffff800a34b0>] autoremove_wake_function+0x9/0x2e
 [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
 [<ffffffff8002dec8>] __wake_up+0x38/0x4f
 [<ffffffff88503e9e>] :emcp:PowerPutSema+0x4e/0x60
 [<ffffffff8006ecd9>] do_gettimeofday+0x40/0x90
 [<ffffffff88735054>] :emcpmpx:MpxTestPath+0x234/0x1fc0
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff887398cb>] :emcpmpx:MpxPeriodicTestPath+0x9b/0x230
 [<ffffffff8006ecd9>] do_gettimeofday+0x40/0x90
 [<ffffffff88739d5a>] :emcpmpx:MpxPeriodicCallout+0x2fa/0x410
 [<ffffffff88739a60>] :emcpmpx:MpxPeriodicCallout+0x0/0x410
 [<ffffffff8850d64d>] :emcp:PowerServiceDaemonQ+0xad/0xd0
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff8850d670>] :emcp:PowerDaemonStart+0x0/0x20
 [<ffffffff8005dfa7>] child_rip+0x0/0x11


Code: 8b 0b c1 e1 14 0b 4b 04 89 ce 89 cf c1 ee 14 81 e7 00 ff 0f 
RIP  [<ffffffff8850ec09>] :emcp:PowerPlatformBottomDispatch+0x329/0x760
 RSP <ffff814072c75a10>

• Verify backtraces of panic task.

Backtraces:

crash> bt
PID: 26035  TASK: ffff81407b484100  CPU: 19  COMMAND: "MpxPeriodicCall"
 #0 [ffff814072c75770] crash_kexec at ffffffff800b099c
 #1 [ffff814072c75830] __die at ffffffff80065137
 #2 [ffff814072c75870] do_page_fault at ffffffff80067484
 #3 [ffff814072c75960] error_exit at ffffffff8005dde9
    [exception RIP: PowerPlatformBottomDispatch+809]   
    RIP: ffffffff8850ec09  RSP: ffff814072c75a10  RFLAGS: 00010206
    RAX: 00000000000001dc  RBX: 0000000000000000  RCX: 00000000000001d0
    RDX: ffff81307be7ded0  RSI: 0000000000000100  RDI: ffff81407d47d008
    RBP: 00000000085001c0   R8: ffff81307fdaa608   R9: 0000000000000000
    R10: ffff812b5e6f5880  R11: 00000000000000e0  R12: ffff8129b9d05788
    R13: 00000000000001dc  R14: ffff812f27b83008  R15: ffff812f27b83080
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
 #4 [ffff814072c75a68] PowerSyncIoBottomDispatch at ffffffff8850f0d8 [emcp]
 #5 [ffff814072c75a98] PowerDispatchX at ffffffff8850f626 [emcp]
 #6 [ffff814072c75ae8] EmsInquiry at ffffffff885107fd [emcp]
 #7 [ffff814072c75b28] SymmetrixKLam_getPathLunStatus at ffffffff8873e966 [emcpmpx]
 #8 [ffff814072c75b58] MpxDefaultTestPath at ffffffff8872d391 [emcpmpx]
 #9 [ffff814072c75b78] MpxLnxTestPath at ffffffff8873cbf3 [emcpmpx]
#10 [ffff814072c75cc8] MpxTestPath at ffffffff88735054 [emcpmpx]
#11 [ffff814072c75e18] MpxPeriodicTestPath at ffffffff887398cb [emcpmpx]
#12 [ffff814072c75e98] MpxPeriodicCallout at ffffffff88739d5a [emcpmpx]
#13 [ffff814072c75f08] PowerServiceDaemonQ at ffffffff8850d64d [emcp]
#14 [ffff814072c75f48] kernel_thread at ffffffff8005dfb1

• Dis-assemble exception pointer: ( PowerPlatformBottomDispatch+809 )

crash> dis -rl PowerPlatformBottomDispatch+809 | tail
0xffffffff8850ebe2 <PowerPlatformBottomDispatch+770>:   shl    $0x4,%ecx
0xffffffff8850ebe5 <PowerPlatformBottomDispatch+773>:   lea    -0x780(%rcx,%rsi,1),%ecx
0xffffffff8850ebec <PowerPlatformBottomDispatch+780>:   mov    %ebp,%eax
0xffffffff8850ebee <PowerPlatformBottomDispatch+782>:   mov    0x18(%rdx),%r8
0xffffffff8850ebf2 <PowerPlatformBottomDispatch+786>:   and    $0xf0,%eax
0xffffffff8850ebf7 <PowerPlatformBottomDispatch+791>:   shr    $0x4,%eax
0xffffffff8850ebfa <PowerPlatformBottomDispatch+794>:   lea    (%rcx,%rax,1),%r13d
0xffffffff8850ebfe <PowerPlatformBottomDispatch+798>:   mov    %r13d,%eax
0xffffffff8850ec01 <PowerPlatformBottomDispatch+801>:   mov    -0x77a649e0(,%rax,8),%rbx
0xffffffff8850ec09 <PowerPlatformBottomDispatch+809>:   mov    (%rbx),%ecx

• Verify unsigned (U) modules are loaded on the system.

crash> mod -t
NAME        LICENSE_GPLOK
oracleasm   40(U)
emcp        41(U)
emcpgpx     41(U)
emcpmpx     41(U)
emcpdm      41(U)
emcpxcrypt  41(U)
emcpvlumd   41(U)

• Check version and srcversion details of emcp module.

crash> mod |grep -i emcp
ffffffff88531d80  emcp               2173088  (not loaded)  [CONFIG_KALLSYMS]
ffffffff8871a480  emcpgpx              55376  (not loaded)  [CONFIG_KALLSYMS]
ffffffff8874c980  emcpmpx             200872  (not loaded)  [CONFIG_KALLSYMS]
ffffffff88761300  emcpdm               75272  (not loaded)  [CONFIG_KALLSYMS]
ffffffff8878b680  emcpxcrypt          166248  (not loaded)  [CONFIG_KALLSYMS]
ffffffff8879d780  emcpvlumd            68064  (not loaded)  [CONFIG_KALLSYMS]

crash> module ffffffff88531d80 | grep -e version
  version = 0x0 <__mod_license952>, 
  srcversion = 0xffff81407e3c8b60 "876E5CA9FC19AC904CD8DC0",