System is crashing in random location with 'soft lockup'
Issue
- System is crashing with soft lockup messages, but the location is different time to time
- It happens in VMWare environment only
Kernel panic - not syncing: softlockup: hung tasks
Pid: 51436, comm: k08agent Not tainted 2.6.32-504.el6.x86_64 #1
Call Trace:
<IRQ> [<ffffffff815292bc>] ? panic+0xa7/0x16f
[<ffffffff810ea0c0>] ? watchdog_timer_fn+0x0/0x1e0
[<ffffffff810ea28a>] ? watchdog_timer_fn+0x1ca/0x1e0
[<ffffffff810a338e>] ? __run_hrtimer+0x8e/0x1a0
[<ffffffff810aa78f>] ? ktime_get_update_offsets+0x4f/0xd0
[<ffffffff810a36f6>] ? hrtimer_interrupt+0xe6/0x260
[<ffffffff8145cac8>] ? __netif_receive_skb+0x6e8/0x750
[<ffffffff810342fd>] ? local_apic_timer_interrupt+0x3d/0x70
[<ffffffff81533c05>] ? smp_apic_timer_interrupt+0x45/0x60
[<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
[<ffffffff8144faed>] ? skb_put+0x2d/0x70
[<ffffffffa00fd5e2>] ? vmxnet3_rq_rx_complete+0x222/0x8d0 [vmxnet3]
[<ffffffff812a2bc0>] ? swiotlb_map_page+0x0/0x100
[<ffffffffa00fde03>] ? vmxnet3_poll_rx_only+0x43/0xc0 [vmxnet3]
[<ffffffff81462083>] ? net_rx_action+0x103/0x2f0
[<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
[<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
[<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
[<ffffffff8100fc15>] ? do_softirq+0x65/0xa0
[<ffffffff8107d765>] ? irq_exit+0x85/0x90
[<ffffffff81533c0a>] ? smp_apic_timer_interrupt+0x4a/0x60
[<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
<EOI> 00 00 00 89 87 c4 00 00 00 e8 d0 1b fd ff 44 8b ab c4 00 00 00 4c 03 ab d0 00 00 00 <41> f6 45 00 01 0f 85 a0 00 00 00 49 8b 84 24 28 02 00 00 48 8b
Call Trace:
<IRQ> [<ffffffffa00fd733>] ? vmxnet3_rq_rx_complete+0x373/0x8d0 [vmxnet3]
[<ffffffff812a2bc0>] ? swiotlb_map_page+0x0/0x100
[<ffffffffa00fde03>] ? vmxnet3_poll_rx_only+0x43/0xc0 [vmxnet3]
[<ffffffff81087d23>] ? run_timer_softirq+0x103/0x340
[<ffffffff81462083>] ? net_rx_action+0x103/0x2f0
[<ffffffff8107d8b1>] ? __do_softirq+0xc1/0x1e0
[<ffffffff810b034a>] ? tick_program_event+0x2a/0x30
[<ffffffff8100c30c>] ? call_softirq+0x1c/0x30
[<ffffffff8100fc15>] ? do_softirq+0x65/0xa0
[<ffffffff8107d765>] ? irq_exit+0x85/0x90
[<ffffffff81533c0a>] ? smp_apic_timer_interrupt+0x4a/0x60
[<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20
<EOI> [<ffffffff8140e420>] ? mdstat_poll+0x0/0x60
[<ffffffff8140e420>] ? mdstat_poll+0x0/0x60
[<ffffffff811f922d>] ? proc_reg_poll+0x6d/0xb0
[<ffffffff811a5f15>] ? do_sys_poll+0x2d5/0x5b0
[<ffffffff811a59f0>] ? __pollwait+0x0/0xf0
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff811a5ae0>] ? pollwake+0x0/0x60
[<ffffffff81014a79>] ? read_tsc+0x9/0x20
[<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
[<ffffffff811a58a5>] ? poll_select_set_timeout+0x95/0xb0
[<ffffffff811a63e1>] ? sys_poll+0x71/0x100
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 5, 6 or 7
- VMWare virtual guest with vmtoolsd installed
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.