Leapp pre-upgrade hangs with multiple stalled tasks in mutex_lock()

Issue

• The leapp preupgrade command hangs with multiple stalled tasks in mutex_lock()

kernel: INFO: task rm:18432 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: rm              D ffff99cf49589660     0 18432   4128 0x00000082
kernel: Call Trace:
kernel: [<ffffffffb93b8bf9>] schedule_preempt_disabled+0x29/0x70
kernel: [<ffffffffb93b6ca7>] __mutex_lock_slowpath+0xc7/0x1e0
kernel: [<ffffffffb8e5ec88>] ? __sb_start_write+0x58/0x120
kernel: [<ffffffffb93b602f>] mutex_lock+0x1f/0x33
kernel: [<ffffffffb8e68231>] lock_rename+0x31/0x100
kernel: [<ffffffffc07c734c>] ovl_lock_rename_workdir+0x1c/0x50 [overlay]
kernel: [<ffffffffc07c9dd0>] ovl_do_remove+0x220/0x420 [overlay]
kernel: [<ffffffffc07c9fe6>] ovl_rmdir+0x16/0x20 [overlay]
kernel: [<ffffffffb8e68b6c>] vfs_rmdir+0xdc/0x150
kernel: [<ffffffffb8e6e711>] do_rmdir+0x1f1/0x220
kernel: [<ffffffffb8e5e13e>] ? ____fput+0xe/0x20
kernel: [<ffffffffb8cc7f70>] ? task_work_run+0xc0/0xe0
kernel: [<ffffffffb8e6f9dd>] SyS_unlinkat+0x2d/0x50
kernel: [<ffffffffb93c539a>] system_call_fastpath+0x25/0x2a
kernel: INFO: task mount:25893 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: mount           D ffff99c8364fc7e0     0 25893  25877 0x00000080
kernel: Call Trace:
kernel: [<ffffffffb93b8bf9>] schedule_preempt_disabled+0x29/0x70
kernel: [<ffffffffb93b6ca7>] __mutex_lock_slowpath+0xc7/0x1e0
kernel: [<ffffffffb93b602f>] mutex_lock+0x1f/0x33
kernel: [<ffffffffb8e68231>] lock_rename+0x31/0x100
kernel: [<ffffffffc07c3dc1>] ovl_fill_super+0x581/0xc00 [overlay]
kernel: [<ffffffffb8ddca64>] ? register_shrinker+0x44/0x50
kernel: [<ffffffffb8e5fc0d>] ? sget+0x7d/0xb0
kernel: [<ffffffffc07c3840>] ? ovl_get_lowerstack.isra.10+0x6a0/0x6a0 [overlay]
kernel: [<ffffffffb8e5fc8f>] mount_nodev+0x4f/0xb0
kernel: [<ffffffffc07c215f>] ovl_mount+0x4f/0x60 [overlay]
kernel: [<ffffffffb8e6083e>] mount_fs+0x3e/0x1c0
kernel: [<ffffffffb8e80327>] vfs_kern_mount+0x67/0x110
kernel: [<ffffffffb8e82b4f>] do_mount+0x1ef/0xd00
kernel: [<ffffffffb8e587fa>] ? __check_object_size+0x1ca/0x250
kernel: [<ffffffffb8e35cfc>] ? kmem_cache_alloc_trace+0x3c/0x200
kernel: [<ffffffffb8e839a3>] SyS_mount+0x83/0xd0
kernel: [<ffffffffb93c539a>] system_call_fastpath+0x25/0x2a

• The system crashed via hung_task panic with the following call traces from the stalled task in mutex_lock().

[ 3240.425503] INFO: task qmgr:2242 blocked for more than 120 seconds.
[ 3240.428110] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3240.430764] qmgr            D ffff8da69ba192c0     0  2242   2227 0x00000080
[ 3240.430775] Call Trace:
[ 3240.430794]  [<ffffffffaadb8bf9>] schedule_preempt_disabled+0x29/0x70
[ 3240.430802]  [<ffffffffaadb6ca7>] __mutex_lock_slowpath+0xc7/0x1e0
[ 3240.430812]  [<ffffffffaadae783>] ? audit_inode.part.23+0x1e/0x24
[ 3240.430818]  [<ffffffffaadb602f>] mutex_lock+0x1f/0x33
[ 3240.430827]  [<ffffffffaa868231>] lock_rename+0x31/0x100
[ 3240.430833]  [<ffffffffaa86ec4f>] SYSC_renameat2+0x22f/0x5a0
[ 3240.430847]  [<ffffffffaa861d11>] ? SYSC_newlstat+0x41/0x70
[ 3240.430852]  [<ffffffffaa86fe9e>] SyS_renameat2+0xe/0x20
[ 3240.430857]  [<ffffffffaa86feee>] SyS_rename+0x1e/0x30
[ 3240.430862]  [<ffffffffaadc539a>] system_call_fastpath+0x25/0x2a
[..]
[ 3240.431286] Kernel panic - not syncing: hung_task: blocked tasks
[ 3240.433869] CPU: 3 PID: 30 Comm: khungtaskd Kdump: loaded Tainted: G ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[ 3240.436185] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 3240.437352] Call Trace:
[ 3240.438329]  [<ffffffffaadb1bec>] dump_stack+0x19/0x1f
[ 3240.439608]  [<ffffffffaadab708>] panic+0xe8/0x21f
[ 3240.439615]  [<ffffffffaa756b6e>] watchdog+0x26e/0x2c0
[ 3240.439619]  [<ffffffffaa756900>] ? reset_hung_task_detector+0x20/0x20
[ 3240.439624]  [<ffffffffaa6cb621>] kthread+0xd1/0xe0
[ 3240.439628]  [<ffffffffaa6cb550>] ? insert_kthread_work+0x40/0x40
[ 3240.439631]  [<ffffffffaadc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[ 3240.439633]  [<ffffffffaa6cb550>] ? insert_kthread_work+0x40/0x40