Container host tasks begin to hang, rtnl mutex held by "fib_sync_down_dev"

Solution Unverified - Updated -

Issue

  • RHEL 8 Docker/Podman container host tasks hang and do not return. Backtraces on the tasks show they are waiting on the rtnl mutex, eg:
crash> bt 1433084
PID: 1433084  TASK: ffffa04497334000  CPU: 20   COMMAND: "snmpd"
 #0 [ffffbe0e6ec1bc40] __schedule at ffffffff91a20881
 #1 [ffffbe0e6ec1bc98] schedule at ffffffff91a20e75
 #2 [ffffbe0e6ec1bcb0] schedule_preempt_disabled at ffffffff91a211ca
 #3 [ffffbe0e6ec1bcb8] __mutex_lock at ffffffff91a22e59
 #4 [ffffbe0e6ec1bd28] dev_ethtool at ffffffff918d6db3
 #5 [ffffbe0e6ec1bd60] dev_ioctl at ffffffff9187a55c
 #6 [ffffbe0e6ec1bd98] sock_do_ioctl at ffffffff91822a44
 #7 [ffffbe0e6ec1be08] sock_ioctl at ffffffff91822d1c
 #8 [ffffbe0e6ec1be80] do_vfs_ioctl at ffffffff91384b74
 #9 [ffffbe0e6ec1bef8] ksys_ioctl at ffffffff913851c4
#10 [ffffbe0e6ec1bf30] __x64_sys_ioctl at ffffffff91385216
#11 [ffffbe0e6ec1bf38] do_syscall_64 at ffffffff9100549b
#12 [ffffbe0e6ec1bf50] entry_SYSCALL_64_after_hwframe at ffffffff91c0012e
  • Investigation of the rtnl mutex holder shows it is a kernel worker task performing netns cleanup in fib_sync_down_dev:
crash> bt 0xffffa01acfca4000
PID: 88894    TASK: ffffa01acfca4000  CPU: 0    COMMAND: "kworker/u80:0"
...
 #5 [ffffbe0e69377c40] fib_sync_down_dev at ffffffff91941730
 #6 [ffffbe0e69377cb0] fib_disable_ip at ffffffff9193ce31
 #7 [ffffbe0e69377cc0] fib_netdev_event at ffffffff9193ea81
 #8 [ffffbe0e69377ce0] notifier_call_chain at ffffffff91121dd7
 #9 [ffffbe0e69377d10] dev_close_many at ffffffff91847175
#10 [ffffbe0e69377d58] rollback_registered_many at ffffffff9184bd78
#11 [ffffbe0e69377db8] unregister_netdevice_many at ffffffff9184c3f7
#12 [ffffbe0e69377dd0] default_device_exit_batch at ffffffff9184c5a4
#13 [ffffbe0e69377e38] cleanup_net at ffffffff9183e144
#14 [ffffbe0e69377e90] process_one_work at ffffffff91119783

Environment

  • Red Hat Enterprise Linux 9.2 or earlier
  • Red Hat Enterprise Linux 8 or earlier
  • RHEL acting as container host, such as Podman or Docker

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content