Task(s) blocked due to CPU's workqueue stalled by hpsa driver

Solution Verified - Updated -

Issue

  • "INFO: task foo:666 blocked for more than 120 seconds." message on console or logs.

  • Tasks stuck while waiting for completion of delayed work.

  • Server hung or panicked with following messages:

    INFO: task sshd:6425 blocked for more than 120 seconds.
          Not tainted 2.6.32-431.11.2.el6.x86_64 #1
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    sshd          D 000000000000000e     0  6425   6418 0x00000080
     ffff88032926f768 0000000000000086 0000000000000000 0000000000000000
     ffff88032926f848 ffffffff8105a924 ffff88032926f7e8 ffff88032926f758
     ffff8803262625f8 ffff88032926ffd8 000000000000fbc8 ffff8803262625f8
    Call Trace:
     [<ffffffff81528dd5>] schedule_timeout+0x215/0x2e0
     [<ffffffff81528a53>] wait_for_common+0x123/0x180
     [<ffffffff81528b6d>] wait_for_completion+0x1d/0x20
     [<ffffffff81095927>] flush_work+0x77/0xc0
     [<ffffffff81095b44>] flush_delayed_work+0x54/0x70
     [<ffffffff8133b525>] tty_flush_to_ldisc+0x15/0x20
     [<ffffffff81336197>] n_tty_poll+0x67/0x1d0
     [<ffffffff81331cda>] tty_poll+0x8a/0xa0
     [<ffffffff811a0c92>] do_select+0x392/0x6c0
     [<ffffffff811a114a>] core_sys_select+0x18a/0x2c0
     [<ffffffff811a14d7>] sys_select+0x47/0x110
     [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
    

    or:

    INFO: task mingetty:5371 blocked for more than 120 seconds.
          Not tainted 2.6.32-504.1.3.el6.x86_64 #1
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    mingetty      D 0000000000000008     0  5371      1 0x00000080
     ffff880439c81b88 0000000000000086 0000000000000000 ffff8804bc549087
     ffff880439c81b78 ffffffff812966a0 0000002debbb7c24 0000000100000001
     ffffffffffffffff 00000000fffe3a75 ffff88043a08b098 ffff880439c81fd8
    Call Trace:
     [<ffffffff8152a8e5>] schedule_timeout+0x215/0x2e0
     [<ffffffff8152a563>] wait_for_common+0x123/0x180
     [<ffffffff8152a67d>] wait_for_completion+0x1d/0x20
     [<ffffffff810983b1>] flush_cpu_workqueue+0x61/0x90
     [<ffffffff81098e84>] flush_workqueue+0x54/0x80
     [<ffffffff81098ec5>] flush_scheduled_work+0x15/0x20
     [<ffffffff813383bc>] tty_ldisc_release+0x3c/0x90
     [<ffffffff813326db>] tty_release_dev+0x40b/0x5e0
     [<ffffffff813328ce>] tty_release+0x1e/0x30
     [<ffffffff8118f8d5>] __fput+0xf5/0x210
     [<ffffffff8118fa15>] fput+0x25/0x30
     [<ffffffff8118ac6d>] filp_close+0x5d/0x90
     [<ffffffff8118ad45>] sys_close+0xa5/0x100
     [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
    

Environment

  • Red Hat Enterprise Linux 6.5
  • Red Hat Enterprise Linux 6.6
  • HP HPSA Driver version 3.4.4-1-RH1 or 3.4.4-1-RH2 included in either one of following packages:
    • kmod-hpsa-3.4.4_1_RH1-1.el6_5.x86_64
    • or kernel-2.6.32-504.el6.x86_64
    • or kernel-2.6.32-504.1.3.el6.x86_64
    • or kernel-2.6.32-504.3.3.el6.x86_64
    • or kernel-2.6.32-504.8.1.el6.x86_64
    • or kernel-2.6.32-504.12.2.el6.x86_64
    • i686 32-bit versions may also be affected
  • HP Smart Array P220i or P420i, with Firmware Version: 2.14, 3.04 or 3.22
  • optionally hp-snmp-agents package (this will trigger the issue faster)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content