Kernel Panic: unable to handle kernel paging request
I'm running RHEL6.8, kernel 2.6.32-642.3.1.el6.x86_64 on VMWare. Over the weekend, one of my production servers restarted due to a kernel panic. I'm not particularly in analyzing core dump files, so I was hoping someone could point me in a useful direction based on what information I have gathered.
Here is the kernel panic message:
BUG: unable to handle kernel paging request at ffff88082bf4e030
IP: [<ffff88082bf4e030>] 0xffff88082bf4e030
PGD 1a8e063 PUD 2e067 PMD 800000082be001e3
Oops: 0011 [#1] SMP
last sysfs file: /sys/devices/system/node/node0/meminfo
CPU 1
Modules linked in: nfsd exportfs autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microc
ode vmware_balloon sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 vmw_pvscsi pata_acpi ata_generic ata_piix vmwgfx ttm drm_kms_helper drm i2c_core dm_mirror dm_r
egion_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 0, comm: swapper Not tainted 2.6.32-642.3.1.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffff88082bf4e030>] [<ffff88082bf4e030>] 0xffff88082bf4e030
RSP: 0018:ffff880028223c68 EFLAGS: 00010097
RAX: ffff88082bf4e000 RBX: 000003e8fde218f7 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000ffffffff RDI: ffff88082bf4e000
RBP: ffff880028223c70 R08: 0000000000000001 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 000000002a7d4f68
R13: ffff88082b3cdb00 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: ffff88082bf4e030 CR3: 00000001a2521000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff880834aac000, task ffff880834aa9520)
Stack:
ffffffff8105eb69 ffff880028223c80 ffffffff810a077f ffff880028223c90
<d> ffffffff8127bbc5 ffff880028223cf0 ffffffff812965c1 ffffffff81b0ecc0
<d> ffffffff81b0ece8 ffffffff81b0ecc0 0000000000000206 ffff880028223d00
Call Trace:
<IRQ>
[<ffffffff8105eb69>] ? __wake_up_common+0x59/0x90
[<ffffffff810a077f>] queue_work+0x1f/0x30
[<ffffffff8127bbc5>] kblockd_schedule_work+0x15/0x20
[<ffffffff812965c1>] cfq_completed_request+0x201/0x7e0
[<ffffffff81276f20>] elv_completed_request+0x60/0x70
[<ffffffff8127e6dc>] __blk_put_request+0x3c/0xd0
[<ffffffff8127e849>] blk_finish_request+0xd9/0x260
[<ffffffff8127edc4>] blk_end_bidi_request+0x54/0x80
[<ffffffff8127ee40>] blk_end_request+0x10/0x20
[<ffffffff813a0800>] scsi_io_completion+0xc0/0x6c0
[<ffffffff8106840c>] ? scheduler_tick+0xcc/0x260
[<ffffffff81396c2d>] scsi_finish_command+0xcd/0x130
[<ffffffff813a0f67>] scsi_softirq_done+0x147/0x170
[<ffffffff81285945>] blk_done_softirq+0x85/0xa0
[<ffffffff81085215>] __do_softirq+0xe5/0x230
[<ffffffff810b8c47>] ? update_ts_time_stats+0x67/0xa0
[<ffffffff8100c38c>] call_softirq+0x1c/0x30
[<ffffffff8100fca5>] do_softirq+0x65/0xa0
[<ffffffff810850a5>] irq_exit+0x85/0x90
[<ffffffff81037655>] smp_call_function_single_interrupt+0x35/0x40
[<ffffffff8100be33>] call_function_single_interrupt+0x13/0x20
<EOI>
[<ffffffff8104601b>] ? native_safe_halt+0xb/0x10
[<ffffffff8101696d>] default_idle+0x4d/0xb0
[<ffffffff81009fe6>] cpu_idle+0xb6/0x110
[<ffffffff815408f9>] start_secondary+0x2c0/0x316
Code: 88 ff ff 30 e0 f4 2b 08 88 ff ff 30 e0 f4 2b 08 88 ff ff 0f 19 e2 fd e8 03 00 00 00 d0 5f 1f 04 00 00 00 30 e0 f4 2b 08 88 ff ff <01> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
RIP [<ffff88082bf4e030>] 0xffff88082bf4e030
RSP <ffff880028223c68>
CR2: ffff88082bf4e030
Backtrace
PID: 0 TASK: ffff880834aa9520 CPU: 1 COMMAND: "swapper"
#0 [ffff880028223850] machine_kexec at ffffffff8103fdcb
#1 [ffff8800282238b0] crash_kexec at ffffffff810d1d62
#2 [ffff880028223980] oops_end at ffffffff8154ba80
#3 [ffff8800282239b0] no_context at ffffffff810518cb
#4 [ffff880028223a00] __bad_area_nosemaphore at ffffffff81051b55
#5 [ffff880028223a50] bad_area_nosemaphore at ffffffff81051c23
#6 [ffff880028223a60] __do_page_fault at ffffffff810523c0
#7 [ffff880028223b80] do_page_fault at ffffffff8154da0e
#8 [ffff880028223bb0] page_fault at ffffffff8154ad15
[exception RIP: unknown or invalid address]
RIP: ffff88082bf4e030 RSP: ffff880028223c68 RFLAGS: 00010097
RAX: ffff88082bf4e000 RBX: 000003e8fde218f7 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000ffffffff RDI: ffff88082bf4e000
RBP: ffff880028223c70 R8: 0000000000000001 R9: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 000000002a7d4f68
R13: ffff88082b3cdb00 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff880028223c68] __wake_up_common at ffffffff8105eb69
#10 [ffff880028223cf8] elv_completed_request at ffffffff81276f20
#11 [ffff880028223d08] __blk_put_request at ffffffff8127e6dc
#12 [ffff880028223d38] blk_finish_request at ffffffff8127e849
#13 [ffff880028223d88] blk_end_bidi_request at ffffffff8127edc4
#14 [ffff880028223db8] blk_end_request at ffffffff8127ee40
#15 [ffff880028223dc8] scsi_io_completion at ffffffff813a0800
#16 [ffff880028223e48] scsi_finish_command at ffffffff81396c2d
#17 [ffff880028223e78] scsi_softirq_done at ffffffff813a0f67
#18 [ffff880028223ea8] blk_done_softirq at ffffffff81285945
#19 [ffff880028223ed8] __do_softirq at ffffffff81085215
#20 [ffff880028223f58] call_softirq at ffffffff8100c38c
#21 [ffff880028223f70] do_softirq at ffffffff8100fca5
#22 [ffff880028223f90] irq_exit at ffffffff810850a5
#23 [ffff880028223fa0] smp_call_function_single_interrupt at ffffffff81037655
#24 [ffff880028223fb0] call_function_single_interrupt at ffffffff8100be33
--- <IRQ stack> ---
#25 [ffff880834aafe28] call_function_single_interrupt at ffffffff8100be33
[exception RIP: native_safe_halt+11]
RIP: ffffffff8104601b RSP: ffff880834aafed8 RFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff880834aafed8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81e03228
RBP: ffffffff8100be2e R8: 0000000000000000 R9: 0000000000000000
R10: 00083559a322bda9 R11: ffff880834aafe68 R12: 0000000000000082
R13: ffff880834aafe48 R14: ffff880028231ac8 R15: ffff880834aafe88
ORIG_RAX: ffffffffffffff04 CS: 0010 SS: 0018
#26 [ffff880834aafee0] default_idle at ffffffff8101696d
#27 [ffff880834aaff00] cpu_idle at ffffffff81009fe6
I found a similar error in the KnowledgeBase, but it seemed to be based around TrendMicro serverprotect, which I am not using.
