The kernel-rt crashes due to kernfs_mutex contention where application threads opening files on sysfs(kernfs) with massive concurrency

Issue

• The kernel-rt crashes due to kernfs_mutex contention where application threads opening files on sysfs(kernfs) with massive concurrency

[76025.127548] INFO: task du:27114 blocked for more than 600 seconds.
[76025.133730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[76025.141555] du              D ffff970989b1de80     0 27114  11841 0x00000080
[76025.148653] Call Trace:
[76025.151112]  [<ffffffff9c780320>] schedule+0x30/0x96
[76025.156084]  [<ffffffff9c780efd>] __rt_mutex_slowlock+0xdd/0x160
[76025.162087]  [<ffffffff9c7813a3>] rt_mutex_slowlock_locked+0xc3/0x1c0
[76025.168525]  [<ffffffff9c78151c>] rt_mutex_slowlock+0x7c/0xc0
[76025.174271]  [<ffffffff9c78160f>] rt_mutex_lock+0x6f/0x90
[76025.179673]  [<ffffffff9c782d6e>] _mutex_lock+0xe/0x10
[76025.184817]  [<ffffffff9c2c9342>] kernfs_iop_permission+0x32/0x60
[76025.190917]  [<ffffffff9c247861>] __inode_permission+0x71/0xe0
[76025.196754]  [<ffffffff9c2478e8>] inode_permission+0x18/0x50
[76025.202413]  [<ffffffff9c249f5e>] link_path_walk+0x27e/0x8c0
[76025.208071]  [<ffffffff9c24c5a5>] ? path_init+0x2d5/0x450
[76025.213469]  [<ffffffff9c24c79a>] path_lookupat+0x7a/0x8e0
[76025.218959]  [<ffffffff9c229830>] ? drain_stock.isra.29+0x70/0x80
[76025.225055]  [<ffffffff9c22db77>] ? __mem_cgroup_try_charge+0x347/0x700
[76025.231669]  [<ffffffff9c24de1f>] ? getname_flags+0x4f/0x1a0
[76025.237329]  [<ffffffff9c24d02b>] filename_lookup+0x2b/0xc0
[76025.242900]  [<ffffffff9c24efb7>] user_path_at_empty+0x67/0xc0
[76025.248735]  [<ffffffff9c1eed35>] ? handle_mm_fault+0x705/0xa60
[76025.254655]  [<ffffffff9c24f021>] user_path_at+0x11/0x20
[76025.259965]  [<ffffffff9c2416b3>] vfs_fstatat+0x63/0xc0
[76025.265191]  [<ffffffff9c241a8e>] SYSC_newstat+0x2e/0x60
[76025.270504]  [<ffffffff9c1349d4>] ? __audit_syscall_entry+0xb4/0x110
[76025.276856]  [<ffffffff9c241f4e>] SyS_newstat+0xe/0x10
[76025.281998]  [<ffffffff9c78d4a8>] tracesys+0xa6/0xcc
   ...
[76025.293199] Kernel panic - not syncing: hung_task: blocked tasks
[76025.293201] CPU: 21 PID: 797 Comm: khungtaskd Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.71.1.rt56.1212.el7.x86_64 #1
[76025.293202] Hardware name: Quanta Cloud Technology Inc. QuantaGrid D52BE-2U 1S5BU9Z003W/S5BE-MB 3UPI (LBG-1G), BIOS 3A11.BT20 09/20/2019
[76025.293202] Call Trace:
[76025.293208]  [<ffffffff9c77ab2d>] dump_stack+0x19/0x1b
[76025.293214]  [<ffffffff9c774e13>] panic+0xe8/0x21f
[76025.293220]  [<ffffffff9c144c10>] watchdog+0x2b0/0x330
[76025.293223]  [<ffffffff9c144960>] ? reset_hung_task_detector+0x20/0x20
[76025.293227]  [<ffffffff9c0b9421>] kthread+0xd1/0xe0
[76025.293231]  [<ffffffff9c0b9350>] ? kthread_worker_fn+0x170/0x170
[76025.293234]  [<ffffffff9c78d077>] ret_from_fork_nospec_begin+0x21/0x21
[76025.293237]  [<ffffffff9c0b9350>] ? kthread_worker_fn+0x170/0x170