RHEL 6 - Kernel panic : Watchdog detected hard LOCKUP on cpu X from __ratelimit when using netconsole.
Issue
- System rebooted with the following visible on the console:
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
Pid: 73163, comm: mcdump Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Call Trace:
<NMI> [<ffffffff8150daac>] ? panic+0xa7/0x16f
[<ffffffff810e121d>] ? watchdog_overflow_callback+0xcd/0xd0
[<ffffffff81117040>] ? __perf_event_overflow+0xb0/0x2a0
[<ffffffff8101a8ad>] ? x86_perf_event_update+0x5d/0xb0
[<ffffffff8101b83d>] ? x86_perf_event_set_period+0xdd/0x170
[<ffffffff81117664>] ? perf_event_overflow+0x14/0x20
[<ffffffff810208d2>] ? intel_pmu_handle_irq+0x192/0x300
[<ffffffff81513686>] ? kprobe_exceptions_notify+0x16/0x430
[<ffffffff815121f9>] ? perf_event_nmi_handler+0x39/0xb0
[<ffffffff81513cb5>] ? notifier_call_chain+0x55/0x80
[<ffffffff81513d1a>] ? atomic_notifier_call_chain+0x1a/0x20
[<ffffffff8109cd1e>] ? notify_die+0x2e/0x30
[<ffffffff8151197b>] ? do_nmi+0x1bb/0x340
[<ffffffff81511240>] ? nmi+0x20/0x30
[<ffffffff8151099f>] ? _spin_lock_irqsave+0x2f/0x40
<<EOE>> <IRQ> [<ffffffff8127d92c>] ? __ratelimit+0x3c/0x110
[<ffffffff8106eb25>] ? printk_ratelimit+0x15/0x20
[<ffffffff8112c1fa>] ? __alloc_pages_nodemask+0x6ca/0x8d0
[<ffffffff81166dc2>] ? kmem_getpages+0x62/0x170
[<ffffffff811679da>] ? fallback_alloc+0x1ba/0x270
[<ffffffff8116742f>] ? cache_grow+0x2cf/0x320
[<ffffffff81167759>] ? ____cache_alloc_node+0x99/0x160
[<ffffffff81168920>] ? kmem_cache_alloc_node_trace+0x90/0x200
[<ffffffff81168b3d>] ? __kmalloc_node+0x4d/0x60
[<ffffffff8143de1d>] ? __alloc_skb+0x6d/0x190
[<ffffffff8143dfa4>] ? __netdev_alloc_skb+0x24/0x50
[<ffffffffa03601c5>] ? efx_fast_push_rx_descriptors+0x345/0x4b0 [sfc]
[<ffffffff8128d630>] ? swiotlb_map_page+0x0/0x100
[<ffffffffa034c23f>] ? efx_process_channel+0x8f/0xc0 [sfc]
[<ffffffffa034c2a3>] ? efx_poll+0x33/0x100 [sfc]
[<ffffffff8144d4c3>] ? net_rx_action+0x103/0x2f0
[<ffffffff810770b1>] ? __do_softirq+0xc1/0x1e0
[<ffffffff810e1760>] ? handle_IRQ_event+0x60/0x170
[<ffffffff8107710f>] ? __do_softirq+0x11f/0x1e0
[<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
[<ffffffff8100de05>] ? do_softirq+0x65/0xa0
[<ffffffff81076e95>] ? irq_exit+0x85/0x90
[<ffffffff81517775>] ? do_IRQ+0x75/0xf0
[<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
<EOI> [<ffffffff81510ab1>] ? _spin_lock+0x21/0x30
[<ffffffffa02889b7>] ? jbd2_journal_stop+0x1e7/0x2b0 [jbd2]
[<ffffffffa0288ec7>] ? start_this_handle+0x97/0x480 [jbd2]
[<ffffffff811ac48b>] ? __mark_inode_dirty+0x3b/0x160
[<ffffffffa0289495>] ? jbd2_journal_start+0xb5/0x100 [jbd2]
[<ffffffffa02ca236>] ? ext4_journal_start_sb+0x56/0xe0 [ext4]
[<ffffffffa02b1e7a>] ? ext4_dirty_inode+0x2a/0x60 [ext4]
[<ffffffff811ac48b>] ? __mark_inode_dirty+0x3b/0x160
[<ffffffff8119c742>] ? file_update_time+0xf2/0x170
[<ffffffff8111c1e0>] ? __generic_file_aio_write+0x230/0x490
[<ffffffff8111c4c8>] ? generic_file_aio_write+0x88/0x100
[<ffffffffa02abfb1>] ? ext4_file_write+0x61/0x1e0 [ext4]
[<ffffffff8100b9ce>] ? common_interrupt+0xe/0x13
[<ffffffff8118109a>] ? do_sync_write+0xfa/0x140
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8121bf26>] ? security_file_permission+0x16/0x20
[<ffffffff81181398>] ? vfs_write+0xb8/0x1a0
[<ffffffff81182679>] ? fget_light+0x19/0x90
[<ffffffff81181c91>] ? sys_write+0x51/0x90
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 6:
- Seen running kernel-2.6.32-358.23.2.el6.x86_64 although other errata kernels are liekly exposed
- Specifically need to be running netconsole
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.