System crashed via hung_task panic from stalled task in truncate_inode_pages_range()

Solution Unverified - Updated -

Issue

  • The system crashed via hung_task panic with the following call traces from stalled task in truncate_inode_pages_range()
[21567716.761385] INFO: task systemd-udevd:7480 blocked for more than 200 seconds.
[21567716.761389] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[21567716.761391] systemd-udevd   D ffff8d3b8e12a6e0     0  7480    825 0x00000086
[21567716.761395] Call Trace:
[21567716.761404]  [<ffffffff8af874c0>] ? bit_wait+0x50/0x50
[21567716.761408]  [<ffffffff8af891b9>] schedule+0x29/0x70
[21567716.761411]  [<ffffffff8af86e81>] schedule_timeout+0x221/0x2d0
[21567716.761418]  [<ffffffff8a906992>] ? ktime_get_ts64+0x52/0xf0
[21567716.761420]  [<ffffffff8af874c0>] ? bit_wait+0x50/0x50
[21567716.761423]  [<ffffffff8af88a6d>] io_schedule_timeout+0xad/0x130
[21567716.761426]  [<ffffffff8af88b08>] io_schedule+0x18/0x20
[21567716.761428]  [<ffffffff8af874d1>] bit_wait_io+0x11/0x50
[21567716.761430]  [<ffffffff8af87081>] __wait_on_bit_lock+0x61/0xc0
[21567716.761435]  [<ffffffff8a9c126e>] ? __find_get_pages+0x11e/0x1c0
[21567716.761438]  [<ffffffff8a9bd7a4>] __lock_page+0x74/0x90
[21567716.761443]  [<ffffffff8a8c7010>] ? wake_bit_function+0x40/0x40
[21567716.761447]  [<ffffffff8a9cf7fb>] truncate_inode_pages_range+0x71b/0x750
[21567716.761451]  [<ffffffff8aa86ab0>] ? __brelse+0x50/0x50
[21567716.761457]  [<ffffffff8ab86bf9>] ? free_cpumask_var+0x9/0x10
[21567716.761461]  [<ffffffff8a859577>] ? native_send_call_func_ipi+0xa7/0x120
[21567716.761465]  [<ffffffff8a9170af>] ? smp_call_function_many+0x1df/0x270
[21567716.761467]  [<ffffffff8aa86a9d>] ? __brelse+0x3d/0x50
[21567716.761470]  [<ffffffff8aa86ab0>] ? __brelse+0x50/0x50
[21567716.761472]  [<ffffffff8aa85e50>] ? mark_buffer_async_write+0x20/0x20
[21567716.761475]  [<ffffffff8ab86bf9>] ? free_cpumask_var+0x9/0x10
[21567716.761477]  [<ffffffff8a91733d>] ? on_each_cpu_cond+0xcd/0x190
[21567716.761495]  [<ffffffff8aa86ab0>] ? __brelse+0x50/0x50
[21567716.761498]  [<ffffffff8a9cf845>] truncate_inode_pages+0x15/0x20
[21567716.761501]  [<ffffffff8aa8e59f>] kill_bdev+0x2f/0x40
[21567716.761504]  [<ffffffff8aa8fd44>] __blkdev_put+0x64/0x1a0
[21567716.761506]  [<ffffffff8aa907fc>] blkdev_put+0x4c/0x130
[21567716.761509]  [<ffffffff8aa90995>] blkdev_close+0x25/0x30
[21567716.761512]  [<ffffffff8aa5088c>] __fput+0xec/0x230
[21567716.761514]  [<ffffffff8aa50abe>] ____fput+0xe/0x10
[21567716.761517]  [<ffffffff8a8c299b>] task_work_run+0xbb/0xe0
[21567716.761521]  [<ffffffff8a8a1954>] do_exit+0x2d4/0xa30
[21567716.761523]  [<ffffffff8a8a212f>] do_group_exit+0x3f/0xa0
[21567716.761528]  [<ffffffff8a8b328e>] get_signal_to_deliver+0x1ce/0x5e0
[21567716.761532]  [<ffffffff8a82c527>] do_signal+0x57/0x6f0
[21567716.761535]  [<ffffffff8a82cc32>] do_notify_resume+0x72/0xc0
[21567716.761538]  [<ffffffff8af962ef>] int_signal+0x12/0x17
[..]
[21567716.761813] Kernel panic - not syncing: hung_task: blocked tasks
[21567716.761863] CPU: 9 PID: 173 Comm: khungtaskd Kdump: loaded Tainted: POE  ------------   3.10.0-1160.53.1.el7.x86_64 #1
[21567716.761938] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 11/23/2021
[21567716.761965] Call Trace:
[21567716.761996]  [<ffffffff8af83579>] dump_stack+0x19/0x1b
[21567716.762020]  [<ffffffff8af7d281>] panic+0xe8/0x21f
[21567716.762043]  [<ffffffff8a94e95e>] watchdog+0x26e/0x2c0
[21567716.763165]  [<ffffffff8a94e6f0>] ? reset_hung_task_detector+0x20/0x20
[21567716.764154]  [<ffffffff8a8c5e61>] kthread+0xd1/0xe0
[21567716.765075]  [<ffffffff8a8c5d90>] ? insert_kthread_work+0x40/0x40
[21567716.766057]  [<ffffffff8af95df7>] ret_from_fork_nospec_begin+0x21/0x21
[21567716.767010]  [<ffffffff8a8c5d90>] ? insert_kthread_work+0x40/0x40

Environment

  • Red Hat Enterpise Linux 7
  • Kernel
  • Multipath

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content