Dead lock occurrs in kmem_cache_destroy_memcg_children
Issue
- A process was blocked in
Dstate atmutex_lock()afterkmem_cache_destory_memcg_children(), and the system crashed withKernel panic - not syncing: hung_task: blocked tasksmessage.
crash> bt 6982
PID: 6982 TASK: ffff8800d2f64e70 CPU: 7 COMMAND: "dockerd"
#0 [ffff8800a9fcfb40] __schedule at ffffffff8168af25
#1 [ffff8800a9fcfba8] schedule_preempt_disabled at ffffffff8168c669
#2 [ffff8800a9fcfbb8] __mutex_lock_slowpath at ffffffff8168a2c5
#3 [ffff8800a9fcfc10] mutex_lock at ffffffff8168972f
#4 [ffff8800a9fcfc28] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
#5 [ffff8800a9fcfc50] kmem_cache_destroy at ffffffff811a56d9
#6 [ffff8800a9fcfc70] bioset_free at ffffffff8123741e
#7 [ffff8800a9fcfc90] __dm_destroy at ffffffffa00029de [dm_mod]
#8 [ffff8800a9fcfcd0] dm_destroy at ffffffffa0003833 [dm_mod]
#9 [ffff8800a9fcfce0] dev_remove at ffffffffa00093de [dm_mod]
#10 [ffff8800a9fcfd10] ctl_ioctl at ffffffffa0009a35 [dm_mod]
#11 [ffff8800a9fcfea8] dm_ctl_ioctl at ffffffffa0009d63 [dm_mod]
#12 [ffff8800a9fcfeb8] do_vfs_ioctl at ffffffff81211ed5
#13 [ffff8800a9fcff30] sys_ioctl at ffffffff81212151
#14 [ffff8800a9fcff80] system_call_fastpath at ffffffff816964c9
RIP: 00007f5a74036567 RSP: 00007f5a1cfd0b40 RFLAGS: 00000293
RAX: 0000000000000010 RBX: ffffffff816964c9 RCX: ffffffffffffffff
RDX: 00007f5a0c01d4a0 RSI: 00000000c138fd04 RDI: 000000000000000a
RBP: 00007f5a74354ef3 R8: 0000000000000001 R9: 0000000000010b60
R10: 00007f5a0c000078 R11: 0000000000000246 R12: 00007f5a1cfd0bd0
R13: 00007f5a0c01d3c0 R14: 00007f5a74354ef3 R15: 00007f5a7435503a
ORIG_RAX: 0000000000000010 CS: 0033 SS: 002b
- A process was blocked in
Dstate atmutex_lock()afterkmem_cache_destroy_memcg_children(). Thekmem_cache_destroy_memcg_childre()was called once again fromkmem_cache_destroy().
crash> bt 0xffff88078aed2f10
PID: 45393 TASK: ffff88078aed2f10 CPU: 1 COMMAND: "kworker/u128:4"
#0 [ffff8800cfb8bba0] __schedule at ffffffff8168af25
#1 [ffff8800cfb8bc08] schedule_preempt_disabled at ffffffff8168c669
#2 [ffff8800cfb8bc18] __mutex_lock_slowpath at ffffffff8168a2c5
#3 [ffff8800cfb8bc70] mutex_lock at ffffffff8168972f
#4 [ffff8800cfb8bc88] kmem_cache_destroy_memcg_children at ffffffff811f3cfe
#5 [ffff8800cfb8bcb0] kmem_cache_destroy at ffffffff811a56d9
#6 [ffff8800cfb8bcd0] kmem_cache_destroy_memcg_children at ffffffff811f3d49
#7 [ffff8800cfb8bcf8] kmem_cache_destroy at ffffffff811a56d9
#8 [ffff8800cfb8bd18] nf_conntrack_cleanup_net_list at ffffffffa034a5db [nf_conntrack]
#9 [ffff8800cfb8bd60] nf_conntrack_pernet_exit at ffffffffa034b45d [nf_conntrack]
#10 [ffff8800cfb8bd88] ops_exit_list at ffffffff815667f3
#11 [ffff8800cfb8bdb8] cleanup_net at ffffffff815678d0
#12 [ffff8800cfb8be20] process_one_work at ffffffff810a7f3b
#13 [ffff8800cfb8be68] worker_thread at ffffffff810a8d76
#14 [ffff8800cfb8bec8] kthread at ffffffff810b052f
#15 [ffff8800cfb8bf50] ret_from_fork at ffffffff81696418
Environment
- Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
