Container host tasks begin to hang, rtnl mutex held by "fib_sync_down_dev"
Issue
- RHEL 8 Docker/Podman container host tasks hang and do not return. Backtraces on the tasks show they are waiting on the rtnl mutex, eg:
crash> bt 1433084
PID: 1433084 TASK: ffffa04497334000 CPU: 20 COMMAND: "snmpd"
#0 [ffffbe0e6ec1bc40] __schedule at ffffffff91a20881
#1 [ffffbe0e6ec1bc98] schedule at ffffffff91a20e75
#2 [ffffbe0e6ec1bcb0] schedule_preempt_disabled at ffffffff91a211ca
#3 [ffffbe0e6ec1bcb8] __mutex_lock at ffffffff91a22e59
#4 [ffffbe0e6ec1bd28] dev_ethtool at ffffffff918d6db3
#5 [ffffbe0e6ec1bd60] dev_ioctl at ffffffff9187a55c
#6 [ffffbe0e6ec1bd98] sock_do_ioctl at ffffffff91822a44
#7 [ffffbe0e6ec1be08] sock_ioctl at ffffffff91822d1c
#8 [ffffbe0e6ec1be80] do_vfs_ioctl at ffffffff91384b74
#9 [ffffbe0e6ec1bef8] ksys_ioctl at ffffffff913851c4
#10 [ffffbe0e6ec1bf30] __x64_sys_ioctl at ffffffff91385216
#11 [ffffbe0e6ec1bf38] do_syscall_64 at ffffffff9100549b
#12 [ffffbe0e6ec1bf50] entry_SYSCALL_64_after_hwframe at ffffffff91c0012e
- Investigation of the rtnl mutex holder shows it is a kernel worker task performing netns cleanup in
fib_sync_down_dev
:
crash> bt 0xffffa01acfca4000
PID: 88894 TASK: ffffa01acfca4000 CPU: 0 COMMAND: "kworker/u80:0"
...
#5 [ffffbe0e69377c40] fib_sync_down_dev at ffffffff91941730
#6 [ffffbe0e69377cb0] fib_disable_ip at ffffffff9193ce31
#7 [ffffbe0e69377cc0] fib_netdev_event at ffffffff9193ea81
#8 [ffffbe0e69377ce0] notifier_call_chain at ffffffff91121dd7
#9 [ffffbe0e69377d10] dev_close_many at ffffffff91847175
#10 [ffffbe0e69377d58] rollback_registered_many at ffffffff9184bd78
#11 [ffffbe0e69377db8] unregister_netdevice_many at ffffffff9184c3f7
#12 [ffffbe0e69377dd0] default_device_exit_batch at ffffffff9184c5a4
#13 [ffffbe0e69377e38] cleanup_net at ffffffff9183e144
#14 [ffffbe0e69377e90] process_one_work at ffffffff91119783
Environment
- Red Hat Enterprise Linux 9.2 or earlier
- Red Hat Enterprise Linux 8 or earlier
- RHEL acting as container host, such as Podman or Docker
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.