RHEL7: kernel panic in shrink_dcache_for_umount_subtree() - BUG: Dentry still in use
Issue
- During a maintenance slot, 4 nodes have crashed the same way. Each time a dynamic library is at the origin of the crash
- Unmounting an NFS4 filesystem crashes with
Dentry ... still in use (1) [unmount of nfs4 ...]
[4319532.467882] BUG: Dentry ffff881020a740c0{i=138c0b,n=libsvml.so} still in use (1) [unmount of nfs4 0:43]
[4319532.477457] ------------[ cut here ]------------
[4319532.482241] kernel BUG at fs/dcache.c:945!
...
PID: 51770 TASK: ffff881ef40f2d80 CPU: 49 COMMAND: "umount.nfs4"
#0 [ffff880da106fad8] machine_kexec at ffffffff8104c4cb
#1 [ffff880da106fb38] crash_kexec at ffffffff810e1fe2
#2 [ffff880da106fc08] oops_end at ffffffff8160c588
#3 [ffff880da106fc30] die at ffffffff8101736b
#4 [ffff880da106fc60] do_trap at ffffffff8160bc60
#5 [ffff880da106fcb0] do_invalid_op at ffffffff810141c4
#6 [ffff880da106fd60] invalid_op at ffffffff816154de
[exception RIP: shrink_dcache_for_umount_subtree+428]
RIP: ffffffff811dd56c RSP: ffff880da106fe10 RFLAGS: 00010246
RAX: 000000000000005b RBX: ffff881020a740c0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88203f22d488 RDI: 0000000000000246
RBP: ffff880da106fe28 R8: 0000000000000096 R9: 00000000000049b3
R10: 0000000000000000 R11: ffff880da106fb26 R12: ffff880fa9aec780
R13: ffffffffa0dbb7a0 R14: ffff8810199e6f40 R15: ffff8810199e6f20
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff880da106fe30] shrink_dcache_for_umount at ffffffff811df109
#8 [ffff880da106fe48] generic_shutdown_super at ffffffff811c8901
#9 [ffff880da106fe70] kill_anon_super at ffffffff811c8ba2
#10 [ffff880da106fe88] nfs_kill_super at ffffffffa0d5726b [nfs]
#11 [ffff880da106fea8] deactivate_locked_super at ffffffff811c8f4d
#12 [ffff880da106fec8] deactivate_super at ffffffff811c9556
#13 [ffff880da106fee0] mntput_no_expire at ffffffff811e6265
#14 [ffff880da106ff08] sys_umount at ffffffff811e739f
#15 [ffff880da106ff80] system_call_fastpath at ffffffff81613da9
RIP: 00007fe5071a9247 RSP: 00007fff94139148 RFLAGS: 00010246
RAX: 00000000000000a6 RBX: ffffffff81613da9 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00007fe50818f080
RBP: 00007fe50818f080 R8: 6e65696c632c3738 R9: 33313d7264646174
R10: 00007fff94138e30 R11: 0000000000000202 R12: 00007fff9413af90
R13: 00007fe50818ef10 R14: 0000000000000001 R15: 00007fe5076b2d38
ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b
- Unmounting an xfs filesystem crashes with
Dentry ... still in use (1) [unmount of xfs dm-8]
[4745627.226278] XFS (sda1): Unmounting Filesystem
[4745627.431235] BUG: Dentry ffff88014e052840{i=9bf8990,n=#ffff88014e053780} still in use (1) [unmount of xfs dm-8]
[4745627.431306] ------------[ cut here ]------------
[4745627.431330] kernel BUG at fs/dcache.c:946!
[4745627.431346] invalid opcode: 0000 [#1] SMP
[4745627.431362] Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc overlay() vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr shpchp parport_pc i2c_piix4 vmw_vmci parport nfsd binfmt_misc nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm ata_piix vmxnet3 libata vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[4745627.431634] CPU: 3 PID: 2816 Comm: umount Tainted: G ------------ T 3.10.0-327.18.2.el7.x86_64 #1
[4745627.431670] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[4745627.431744] task: ffff8800bb495c00 ti: ffff880328d00000 task.ti: ffff880328d00000
[4745627.431779] RIP: 0010:[<ffffffff811f5bdc>] [<ffffffff811f5bdc>] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[4745627.431830] RSP: 0018:ffff880328d03e28 EFLAGS: 00010246
[4745627.431845] RAX: 0000000000000062 RBX: ffff88014e052840 RCX: 0000000000000000
[4745627.431866] RDX: 0000000000000000 RSI: ffff88033fd8d6c8 RDI: 0000000000000246
[4745627.431886] RBP: ffff880328d03e40 R08: 0000000000000086 R09: 00000000000016b7
[4745627.431907] R10: 666f20746e756f6d R11: 2d6d642073667820 R12: ffff88014e053900
[4745627.431927] R13: ffffffffa02549a0 R14: ffff880036bb6d40 R15: ffff880036bb6d20
[4745627.431956] FS: 00007ff787911880(0000) GS:ffff88033fd80000(0000) knlGS:0000000000000000
[4745627.431994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4745627.432020] CR2: 00007ff7874cdb70 CR3: 0000000328223000 CR4: 00000000000007e0
[4745627.432084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4745627.432139] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4745627.432160] Stack:
[4745627.432167] ffff88032dbd4320 ffff88032dbd4000 0000000000000083 ffff880328d03e58
[4745627.432192] ffffffff811f77bf ffff88032dbd4000 ffff880328d03e80 ffffffff811e0c71
[4745627.432216] ffff8800365f5380 0000000000000083 ffff880036bb6d20 ffff880328d03ea0
[4745627.432240] Call Trace:
[4745627.432252] [<ffffffff811f77bf>] shrink_dcache_for_umount+0x2f/0x60
[4745627.432274] [<ffffffff811e0c71>] generic_shutdown_super+0x21/0xe0
[4745627.432293] [<ffffffff811e10f7>] kill_block_super+0x27/0x70
[4745627.432314] [<ffffffff811e1439>] deactivate_locked_super+0x49/0x60
[4745627.432333] [<ffffffff811e1a36>] deactivate_super+0x46/0x60
[4745627.432353] [<ffffffff811fe9a5>] mntput_no_expire+0xc5/0x120
[4745627.432372] [<ffffffff811ffadf>] SyS_umount+0x9f/0x3c0
[4745627.432401] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
[4745627.432420] Code: 00 00 48 8b 40 28 4c 8b 08 48 8b 43 30 48 85 c0 74 1b 48 8b 50 40 48 89 34 24 48 c7 c7 c8 a5 87 81 48 89 de 31 c0 e8 1c 98 43 00 <0f> 0b 31 d2 eb e5 0f 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 66
[4745627.432554] RIP [<ffffffff811f5bdc>] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[4745627.432578] RSP <ffff880328d03e28>
[4745627.434488] ---[ end trace facbb9d0e6cff0de ]---
[4745627.434502] Kernel panic - not syncing: Fatal exception
Environment
- Red Hat Entreprise Linux 7
- kernel prior to 3.10.0-514.el7
- NFS4 client or Docker(overlayfs)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.