A great number of tasks calling percpu_rwsem_wait() or __rt_mutex_slowlock() are getting stuck for a long time

Issue

• A great number of tasks calling percpu_rwsem_wait() or __rt_mutex_slowlock() are getting stuck for a long time

    [...]
[288914.308281] INFO: task systemd:1 blocked for more than 600 seconds.
[288914.308287]       Tainted: G        W  OE    --------- -  - 4.18.0-372.69.1.rt7.227.el8_6.x86_64 #1
[288914.308289] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[288914.308290] task:systemd         state:D stack:    0 pid:    1 ppid:     0 flags:0x00080000
[288914.308295] Call Trace:
[288914.308299]  __schedule+0x37b/0x8e0
[288914.308307]  ? proc_cgroup_show+0x30/0x290
[288914.308313]  schedule+0x6c/0x120
[288914.308316]  rt_mutex_slowlock_block.isra.18+0x9c/0x170
[288914.308320]  rt_mutex_slowlock.constprop.21+0xd4/0x140
[288914.308324]  proc_cgroup_show+0x4a/0x290
[288914.308328]  proc_single_show+0x4e/0xa0
[288914.308333]  seq_read+0x163/0x420
[288914.308370]  vfs_read+0x91/0x150
[288914.308375]  ksys_read+0x52/0xc0
[288914.308377]  do_syscall_64+0x87/0x1b0
[288914.308382]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[288914.308386] RIP: 0033:0x7fa1f4b2c534
[288914.308394] Code: Unable to access opcode bytes at RIP 0x7fa1f4b2c50a.
[288914.308395] RSP: 002b:00007fff5eb59870 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[288914.308398] RAX: ffffffffffffffda RBX: 000000000000002c RCX: 00007fa1f4b2c534
[288914.308400] RDX: 0000000000000400 RSI: 0000559669ed6bf0 RDI: 000000000000002c
[288914.308401] RBP: 0000559669ed6bf0 R08: 0000000000000000 R09: 0000000000000000
[288914.308402] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400
[288914.308403] R13: 00007fa1f4dc83a0 R14: 00000000000007ff R15: 000055966a027340
[288914.308527] INFO: task systemd-journal:1540 blocked for more than 600 seconds.
[288914.308529]       Tainted: G        W  OE    --------- -  - 4.18.0-372.69.1.rt7.227.el8_6.x86_64 #1
[288914.308530] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[288914.308531] task:systemd-journal state:D stack:    0 pid: 1540 ppid:     1 flags:0x00080100
[288914.308534] Call Trace:
[288914.308536]  __schedule+0x37b/0x8e0
[288914.308549]  schedule+0x6c/0x120
[288914.308551]  rt_mutex_slowlock_block.isra.18+0x9c/0x170
[288914.308555]  rt_mutex_slowlock.constprop.21+0xd4/0x140
[288914.308559]  proc_cgroup_show+0x4a/0x290
[288914.308562]  proc_single_show+0x4e/0xa0
[288914.308565]  seq_read+0x163/0x420
[288914.308568]  vfs_read+0x91/0x150
[288914.308571]  ksys_read+0x52/0xc0
[288914.308573]  do_syscall_64+0x87/0x1b0
[288914.308576]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[288914.308578] RIP: 0033:0x7f398e1d1534
[288914.308581] Code: Unable to access opcode bytes at RIP 0x7f398e1d150a.
[288914.308582] RSP: 002b:00007ffd9911bbe0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[288914.308583] RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007f398e1d1534
[288914.308585] RDX: 0000000000000400 RSI: 000055c3e20608b0 RDI: 0000000000000018
[288914.308586] RBP: 000055c3e20608b0 R08: 0000000000000000 R09: 0000000000000000
[288914.308587] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400
[288914.308588] R13: 00007f398e46d3a0 R14: 00000000000007ff R15: 000055c3e2003030
[288914.308598] INFO: task ovs-vswitchd:2285 blocked for more than 600 seconds.
[288914.308600]       Tainted: G        W  OE    --------- -  - 4.18.0-372.69.1.rt7.227.el8_6.x86_64 #1
[288914.308601] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[288914.308601] task:ovs-vswitchd    state:D stack:    0 pid: 2285 ppid:     1 flags:0x00080080
[288914.308604] Call Trace:
[288914.308605]  __schedule+0x37b/0x8e0
[288914.308608]  schedule+0x6c/0x120
[288914.308611]  percpu_rwsem_wait+0x13f/0x170
[288914.308615]  ? __percpu_rwsem_trylock.part.7+0x70/0x70
[288914.308619]  __percpu_down_read+0x64/0x90
[288914.308622]  cgroup_can_fork+0x1e3/0x220
[288914.308626]  copy_process+0x1232/0x1d30
[288914.308633]  _do_fork+0x8b/0x3a0
[288914.308637]  ? syscall_trace_enter+0x1db/0x300
[288914.308640]  ? __audit_syscall_exit+0x258/0x2c0
[288914.308644]  do_syscall_64+0x87/0x1b0
[288914.308647]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[288914.308649] RIP: 0033:0x7f4e94bfcdc5
[288914.308652] Code: Unable to access opcode bytes at RIP 0x7f4e94bfcd9b.
[288914.308652] RSP: 002b:00007ffe2b582f98 EFLAGS: 00003206 ORIG_RAX: 0000000000000038
[288914.308654] RAX: ffffffffffffffda RBX: 00007f4e96e5ec40 RCX: 00007f4e94bfcdc5
[288914.308655] RDX: 00007f4e96e5ef10 RSI: 00007f4e96e5cff0 RDI: 00000000003d0f00
[288914.308657] RBP: 00007ffe2b5830f0 R08: 00007f4e96e5ec40 R09: 00007f4e96e5ec40
[288914.308658] R10: 00007f4e96e5ef10 R11: 0000000000003206 R12: 00007ffe2b58303e
[288914.308659] R13: 00007ffe2b58303f R14: 00007ffe2b5830f0 R15: 00007f4e96e5d000
    [...]