System rebooted due to blocked tasks
Issue
-
Frequent reboot (unexpected). Trending show system rebooted once every 2-3 days, usually at night around 12 - 3 am (GMT+0800). No abnormalities found from log. No hardware errors. No application errors. System is in cloud environment.
-
Kernel log shows:
INFO: task pdflush:561 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pdflush D ffff810009025e20 0 561 163 562 560 (L-TLB)
ffff81043e52db00 0000000000000046 0000000000bc0368 ffff81043cd93258
0000000000000286 000000000000000a ffff81043fb45830 ffff81010eff17f0
000571522ba70e53 0000000000009046 ffff81043fb45a18 000000048815c9a4
Call Trace:
[<ffffffff8803200f>] :jbd:start_this_handle+0x2e9/0x370
[<ffffffff800a3ccf>] autoremove_wake_function+0x0/0x2e
[<ffffffff800dff1d>] alternate_node_alloc+0x70/0x8c
[<ffffffff88032161>] :jbd:journal_start+0xcb/0x102
[<ffffffff8805077c>] :ext3:ext3_ordered_writepage+0x7f/0x19c
[<ffffffff8001d5b5>] mpage_writepages+0x1bf/0x37d
[<ffffffff880506fd>] :ext3:ext3_ordered_writepage+0x0/0x19c
[<ffffffff80157f06>] __next_cpu+0x19/0x28
[<ffffffff8008dfa7>] find_busiest_group+0x20d/0x621
[<ffffffff8005aed0>] do_writepages+0x29/0x2f
[<ffffffff8002ff06>] __writeback_single_inode+0x1a2/0x31c
[<ffffffff8002e4b9>] __wake_up+0x38/0x4f
[<ffffffff80021591>] sync_sb_inodes+0x1b7/0x271
[<ffffffff800a3ab7>] keventd_create_kthread+0x0/0xc4
[<ffffffff800512f0>] writeback_inodes+0x82/0xd8
[<ffffffff800cda1d>] wb_kupdate+0xf0/0x16a
[<ffffffff80056897>] pdflush+0x0/0x1fb
[<ffffffff800569e8>] pdflush+0x151/0x1fb
[<ffffffff800cd92d>] wb_kupdate+0x0/0x16a
[<ffffffff80032c23>] kthread+0xfe/0x132
[<ffffffff8005dfc1>] child_rip+0xa/0x11
[<ffffffff800a3ab7>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032b25>] kthread+0x0/0x132
[<ffffffff8005dfb7>] child_rip+0x0/0x11
Kernel panic - not syncing: hung_task: blocked tasks
Environment
- Red hat Enterprise Linux 5, 6
- OS running in a VM
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.