RHEL7: kernel panic in shrink_dcache_for_umount_subtree() - BUG: Dentry still in use

Solution Verified - Updated -

Issue

  • During a maintenance slot, 4 nodes have crashed the same way. Each time a dynamic library is at the origin of the crash
  • Unmounting an NFS4 filesystem crashes with Dentry ... still in use (1) [unmount of nfs4 ...]
[4319532.467882] BUG: Dentry ffff881020a740c0{i=138c0b,n=libsvml.so} still in use (1) [unmount of nfs4 0:43]
[4319532.477457] ------------[ cut here ]------------
[4319532.482241] kernel BUG at fs/dcache.c:945!
...

PID: 51770 TASK: ffff881ef40f2d80 CPU: 49 COMMAND: "umount.nfs4"
 #0 [ffff880da106fad8] machine_kexec at ffffffff8104c4cb
 #1 [ffff880da106fb38] crash_kexec at ffffffff810e1fe2
 #2 [ffff880da106fc08] oops_end at ffffffff8160c588
 #3 [ffff880da106fc30] die at ffffffff8101736b
 #4 [ffff880da106fc60] do_trap at ffffffff8160bc60
 #5 [ffff880da106fcb0] do_invalid_op at ffffffff810141c4
 #6 [ffff880da106fd60] invalid_op at ffffffff816154de
    [exception RIP: shrink_dcache_for_umount_subtree+428]
    RIP: ffffffff811dd56c RSP: ffff880da106fe10 RFLAGS: 00010246
    RAX: 000000000000005b RBX: ffff881020a740c0 RCX: 0000000000000000
    RDX: 0000000000000000 RSI: ffff88203f22d488 RDI: 0000000000000246
    RBP: ffff880da106fe28 R8: 0000000000000096 R9: 00000000000049b3
    R10: 0000000000000000 R11: ffff880da106fb26 R12: ffff880fa9aec780
    R13: ffffffffa0dbb7a0 R14: ffff8810199e6f40 R15: ffff8810199e6f20
    ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
 #7 [ffff880da106fe30] shrink_dcache_for_umount at ffffffff811df109
 #8 [ffff880da106fe48] generic_shutdown_super at ffffffff811c8901
 #9 [ffff880da106fe70] kill_anon_super at ffffffff811c8ba2
#10 [ffff880da106fe88] nfs_kill_super at ffffffffa0d5726b [nfs]
#11 [ffff880da106fea8] deactivate_locked_super at ffffffff811c8f4d
#12 [ffff880da106fec8] deactivate_super at ffffffff811c9556
#13 [ffff880da106fee0] mntput_no_expire at ffffffff811e6265
#14 [ffff880da106ff08] sys_umount at ffffffff811e739f
#15 [ffff880da106ff80] system_call_fastpath at ffffffff81613da9
    RIP: 00007fe5071a9247 RSP: 00007fff94139148 RFLAGS: 00010246
    RAX: 00000000000000a6 RBX: ffffffff81613da9 RCX: 0000000000000000
    RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00007fe50818f080
    RBP: 00007fe50818f080 R8: 6e65696c632c3738 R9: 33313d7264646174
    R10: 00007fff94138e30 R11: 0000000000000202 R12: 00007fff9413af90
    R13: 00007fe50818ef10 R14: 0000000000000001 R15: 00007fe5076b2d38
    ORIG_RAX: 00000000000000a6 CS: 0033 SS: 002b
  • Unmounting an xfs filesystem crashes with Dentry ... still in use (1) [unmount of xfs dm-8]
[4745627.226278] XFS (sda1): Unmounting Filesystem
[4745627.431235] BUG: Dentry ffff88014e052840{i=9bf8990,n=#ffff88014e053780} still in use (1) [unmount of xfs dm-8]
[4745627.431306] ------------[ cut here ]------------
[4745627.431330] kernel BUG at fs/dcache.c:946!
[4745627.431346] invalid opcode: 0000 [#1] SMP 
[4745627.431362] Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack bridge stp llc overlay() vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr shpchp parport_pc i2c_piix4 vmw_vmci parport nfsd binfmt_misc nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm ata_piix vmxnet3 libata vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[4745627.431634] CPU: 3 PID: 2816 Comm: umount Tainted: G               ------------ T 3.10.0-327.18.2.el7.x86_64 #1
[4745627.431670] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[4745627.431744] task: ffff8800bb495c00 ti: ffff880328d00000 task.ti: ffff880328d00000
[4745627.431779] RIP: 0010:[<ffffffff811f5bdc>]  [<ffffffff811f5bdc>] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[4745627.431830] RSP: 0018:ffff880328d03e28  EFLAGS: 00010246
[4745627.431845] RAX: 0000000000000062 RBX: ffff88014e052840 RCX: 0000000000000000
[4745627.431866] RDX: 0000000000000000 RSI: ffff88033fd8d6c8 RDI: 0000000000000246
[4745627.431886] RBP: ffff880328d03e40 R08: 0000000000000086 R09: 00000000000016b7
[4745627.431907] R10: 666f20746e756f6d R11: 2d6d642073667820 R12: ffff88014e053900
[4745627.431927] R13: ffffffffa02549a0 R14: ffff880036bb6d40 R15: ffff880036bb6d20
[4745627.431956] FS:  00007ff787911880(0000) GS:ffff88033fd80000(0000) knlGS:0000000000000000
[4745627.431994] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4745627.432020] CR2: 00007ff7874cdb70 CR3: 0000000328223000 CR4: 00000000000007e0
[4745627.432084] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4745627.432139] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4745627.432160] Stack:
[4745627.432167]  ffff88032dbd4320 ffff88032dbd4000 0000000000000083 ffff880328d03e58
[4745627.432192]  ffffffff811f77bf ffff88032dbd4000 ffff880328d03e80 ffffffff811e0c71
[4745627.432216]  ffff8800365f5380 0000000000000083 ffff880036bb6d20 ffff880328d03ea0
[4745627.432240] Call Trace:
[4745627.432252]  [<ffffffff811f77bf>] shrink_dcache_for_umount+0x2f/0x60
[4745627.432274]  [<ffffffff811e0c71>] generic_shutdown_super+0x21/0xe0
[4745627.432293]  [<ffffffff811e10f7>] kill_block_super+0x27/0x70
[4745627.432314]  [<ffffffff811e1439>] deactivate_locked_super+0x49/0x60
[4745627.432333]  [<ffffffff811e1a36>] deactivate_super+0x46/0x60
[4745627.432353]  [<ffffffff811fe9a5>] mntput_no_expire+0xc5/0x120
[4745627.432372]  [<ffffffff811ffadf>] SyS_umount+0x9f/0x3c0
[4745627.432401]  [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
[4745627.432420] Code: 00 00 48 8b 40 28 4c 8b 08 48 8b 43 30 48 85 c0 74 1b 48 8b 50 40 48 89 34 24 48 c7 c7 c8 a5 87 81 48 89 de 31 c0 e8 1c 98 43 00 <0f> 0b 31 d2 eb e5 0f 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 
[4745627.432554] RIP  [<ffffffff811f5bdc>] shrink_dcache_for_umount_subtree+0x1ac/0x1c0
[4745627.432578]  RSP <ffff880328d03e28>
[4745627.434488] ---[ end trace facbb9d0e6cff0de ]---
[4745627.434502] Kernel panic - not syncing: Fatal exception

Environment

  • Red Hat Entreprise Linux 7
  • kernel prior to 3.10.0-514.el7
  • NFS4 client or Docker(overlayfs)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content