Task(s) blocked due to CPU's workqueue stalled by hpsa driver
Issue
-
"INFO: task foo:666 blocked for more than 120 seconds." message on console or logs.
-
Tasks stuck while waiting for completion of delayed work.
-
Server hung or panicked with following messages:
INFO: task sshd:6425 blocked for more than 120 seconds. Not tainted 2.6.32-431.11.2.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. sshd D 000000000000000e 0 6425 6418 0x00000080 ffff88032926f768 0000000000000086 0000000000000000 0000000000000000 ffff88032926f848 ffffffff8105a924 ffff88032926f7e8 ffff88032926f758 ffff8803262625f8 ffff88032926ffd8 000000000000fbc8 ffff8803262625f8 Call Trace: [<ffffffff81528dd5>] schedule_timeout+0x215/0x2e0 [<ffffffff81528a53>] wait_for_common+0x123/0x180 [<ffffffff81528b6d>] wait_for_completion+0x1d/0x20 [<ffffffff81095927>] flush_work+0x77/0xc0 [<ffffffff81095b44>] flush_delayed_work+0x54/0x70 [<ffffffff8133b525>] tty_flush_to_ldisc+0x15/0x20 [<ffffffff81336197>] n_tty_poll+0x67/0x1d0 [<ffffffff81331cda>] tty_poll+0x8a/0xa0 [<ffffffff811a0c92>] do_select+0x392/0x6c0 [<ffffffff811a114a>] core_sys_select+0x18a/0x2c0 [<ffffffff811a14d7>] sys_select+0x47/0x110 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
or:
INFO: task mingetty:5371 blocked for more than 120 seconds. Not tainted 2.6.32-504.1.3.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. mingetty D 0000000000000008 0 5371 1 0x00000080 ffff880439c81b88 0000000000000086 0000000000000000 ffff8804bc549087 ffff880439c81b78 ffffffff812966a0 0000002debbb7c24 0000000100000001 ffffffffffffffff 00000000fffe3a75 ffff88043a08b098 ffff880439c81fd8 Call Trace: [<ffffffff8152a8e5>] schedule_timeout+0x215/0x2e0 [<ffffffff8152a563>] wait_for_common+0x123/0x180 [<ffffffff8152a67d>] wait_for_completion+0x1d/0x20 [<ffffffff810983b1>] flush_cpu_workqueue+0x61/0x90 [<ffffffff81098e84>] flush_workqueue+0x54/0x80 [<ffffffff81098ec5>] flush_scheduled_work+0x15/0x20 [<ffffffff813383bc>] tty_ldisc_release+0x3c/0x90 [<ffffffff813326db>] tty_release_dev+0x40b/0x5e0 [<ffffffff813328ce>] tty_release+0x1e/0x30 [<ffffffff8118f8d5>] __fput+0xf5/0x210 [<ffffffff8118fa15>] fput+0x25/0x30 [<ffffffff8118ac6d>] filp_close+0x5d/0x90 [<ffffffff8118ad45>] sys_close+0xa5/0x100 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 6.5
- Red Hat Enterprise Linux 6.6
- HP HPSA Driver version 3.4.4-1-RH1 or 3.4.4-1-RH2 included in either one of following packages:
- kmod-hpsa-3.4.4_1_RH1-1.el6_5.x86_64
- or kernel-2.6.32-504.el6.x86_64
- or kernel-2.6.32-504.1.3.el6.x86_64
- or kernel-2.6.32-504.3.3.el6.x86_64
- or kernel-2.6.32-504.8.1.el6.x86_64
- or kernel-2.6.32-504.12.2.el6.x86_64
- i686 32-bit versions may also be affected
- HP Smart Array P220i or P420i, with Firmware Version: 2.14, 3.04 or 3.22
- optionally hp-snmp-agents package (this will trigger the issue faster)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.