Oracle RAC node gets crashed with errors 'Kernel panic - not syncing: hung_task: blocked task' due to CPU blocking realtime task

Solution Verified - Updated -

Issue

  • Oracle RAC node gets crashed with errors 'Kernel panic - not syncing: hung_task: blocked task'

    INFO: task jbd2/dm-48-8:11406 blocked for more than 360 seconds.
          Tainted: P           -- ------------    2.6.32-573.22.1.el6.x86_64 #1
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    jbd2/dm-48-8  D 000000000000002e     0 11406      2 0x00000000
     ffff88801d1b3c20 0000000000000046 ffff88801d1b3b80 ffffffff8126ff24
     ffff88401d9b2b80 ffff88401da28f00 ffff88801d1b3bf0 ffffffffa0004d9f
     ffff88801d1b3bf0 0000000000000000 ffff88801ef525f8 ffff88801d1b3fd8
    Call Trace:
     [<ffffffff8126ff24>] ? blk_unplug+0x34/0x70
     [<ffffffffa0004d9f>] ? dm_table_unplug_all+0x5f/0x100 [dm_mod]
     [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50
     [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50
     [<ffffffff81539b33>] io_schedule+0x73/0xc0
     [<ffffffff811c9600>] sync_buffer+0x40/0x50
     [<ffffffff8153a61f>] __wait_on_bit+0x5f/0x90
     [<ffffffff811c95c0>] ? sync_buffer+0x0/0x50
     [<ffffffff8153a6c8>] out_of_line_wait_on_bit+0x78/0x90
     [<ffffffff810a18a0>] ? wake_bit_function+0x0/0x50
     [<ffffffff810a16a7>] ? bit_waitqueue+0x17/0xd0
     [<ffffffff811c95b6>] __wait_on_buffer+0x26/0x30
     [<ffffffffa03d9116>] jbd2_journal_commit_transaction+0xaa6/0x14f0 [jbd2]
     [<ffffffff8108aa8b>] ? try_to_del_timer_sync+0x7b/0xe0
     [<ffffffffa03dea38>] kjournald2+0xb8/0x220 [jbd2]
     [<ffffffff810a1820>] ? autoremove_wake_function+0x0/0x40
     [<ffffffffa03de980>] ? kjournald2+0x0/0x220 [jbd2]
     [<ffffffff810a138e>] kthread+0x9e/0xc0
     [<ffffffff8100c28a>] child_rip+0xa/0x20
     [<ffffffff810a12f0>] ? kthread+0x0/0xc0
     [<ffffffff8100c280>] ? child_rip+0x0/0x20
    Kernel panic - not syncing: hung_task: blocked tasks
    Pid: 851, comm: khungtaskd Tainted: P           -- ------------    2.6.32-573.22.1.el6.x86_64 #1
    Call Trace:
     [<ffffffff81538c41>] ? panic+0xa7/0x16f
     [<ffffffff810ed114>] ? watchdog+0x254/0x290
    [...]
    

Environment

  • Red Hat Enterprise Linux 6
  • Oracle RAC

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content