Oracle RAC node gets crashed with errors 'Kernel panic - not syncing: hung_task: blocked task' due to CPU blocking realtime task
Issue
-
Oracle RAC node gets crashed with errors 'Kernel panic - not syncing: hung_task: blocked task'
INFO: task jbd2/dm-48-8:11406 blocked for more than 360 seconds. Tainted: P -- ------------ 2.6.32-573.22.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. jbd2/dm-48-8 D 000000000000002e 0 11406 2 0x00000000 ffff88801d1b3c20 0000000000000046 ffff88801d1b3b80 ffffffff8126ff24 ffff88401d9b2b80 ffff88401da28f00 ffff88801d1b3bf0 ffffffffa0004d9f ffff88801d1b3bf0 0000000000000000 ffff88801ef525f8 ffff88801d1b3fd8 Call Trace: [<ffffffff8126ff24>] ? blk_unplug+0x34/0x70 [<ffffffffa0004d9f>] ? dm_table_unplug_all+0x5f/0x100 [dm_mod] [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50 [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50 [<ffffffff81539b33>] io_schedule+0x73/0xc0 [<ffffffff811c9600>] sync_buffer+0x40/0x50 [<ffffffff8153a61f>] __wait_on_bit+0x5f/0x90 [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50 [<ffffffff8153a6c8>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff810a18a0>] ? wake_bit_function+0x0/0x50 [<ffffffff810a16a7>] ? bit_waitqueue+0x17/0xd0 [<ffffffff811c95b6>] __wait_on_buffer+0x26/0x30 [<ffffffffa03d9116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2] [<ffffffff8108aa8b>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffffa03dea38>] kjournald2+0xb8/0x220 [jbd2] [<ffffffff810a1820>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa03de980>] ? kjournald2+0x0/0x220 [jbd2] [<ffffffff810a138e>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a12f0>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 Kernel panic - not syncing: hung_task: blocked tasks Pid: 851, comm: khungtaskd Tainted: P -- ------------ 2.6.32-573.22.1.el6.x86_64 #1 Call Trace: [<ffffffff81538c41>] ? panic+0xa7/0x16f [<ffffffff810ed114>] ? watchdog+0x254/0x290 [...]
Environment
- Red Hat Enterprise Linux 6
- Oracle RAC
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.