VM panics with blocked tasks due to slow or stalled I/O

Solution Unverified - Updated -

Issue

  • System rebooted due to blocked tasks
  • Frequent reboot (unexpected). Trending show system rebooted once every 2-3 days, usually at night around 12 - 3 am (GMT+0800). No abnormalities found from log. No hardware errors. No application errors. System is in cloud environment.
  • VM guest panicked with the following:
INFO: task kjournald:911 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald     D ffff81000901d7a0     0   911    279           936   873 (L-TLB)
 ffff81083de83cf0 0000000000000046 ffff81083fafc080 ffff81083e0cc540
 0000000dd7e412dc 000000000000000a ffff81083fafc080 ffff81011cfc8080
 000000115554a5db 00000000000020cd ffff81083fafc268 000000033e6daa38
Call Trace:
 [<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
 [<ffffffff800154ce>] sync_buffer+0x0/0x3f
 [<ffffffff800647ea>] io_schedule+0x3f/0x67
 [<ffffffff80015509>] sync_buffer+0x3b/0x3f
 [<ffffffff80064a16>] __wait_on_bit+0x40/0x6e
 [<ffffffff800154ce>] sync_buffer+0x0/0x3f
 [<ffffffff80064ab0>] out_of_line_wait_on_bit+0x6c/0x78
 [<ffffffff800a1bd2>] wake_bit_function+0x0/0x23
 [<ffffffff880339a5>] :jbd:journal_commit_transaction+0x543/0x1066
 [<ffffffff8003ddd5>] lock_timer_base+0x1b/0x3c
 [<ffffffff8004b6b6>] try_to_del_timer_sync+0x7f/0x88
 [<ffffffff880375d3>] :jbd:kjournald+0xc1/0x213
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff800a198c>] keventd_create_kthread+0x0/0xc4
 [<ffffffff88037512>] :jbd:kjournald+0x0/0x213
 [<ffffffff800a198c>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032bdc>] kthread+0xfe/0x132
 [<ffffffff8005efb1>] child_rip+0xa/0x11
 [<ffffffff800a198c>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032ade>] kthread+0x0/0x132
 [<ffffffff8005efa7>] child_rip+0x0/0x11

Kernel panic - not syncing: hung_task: blocked tasks

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • VM

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.