RHEL7: NFS4 server kernel panic with RIP locks_in_grace called from nfsd4_process_open2

Issue

• Prior to the panic, a number of messages are seen indicating an invalid response was received on a NFS4 callback channel, receive_cb_reply: Got unrecognized reply, and client 1.2.3.4 testing state ID with incorrect client ID

[ 6588.914463] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff880231a91800 xid 8aa8279c
[ 6644.081946] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff880231a91800 xid 8aa82811
[ 6662.894826] NFSD: client 1.2.3.4 testing state ID with incorrect client ID
[ 6663.557121] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff8801abcb5000 xid e823de78
[ 6678.944874] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff88007869f800 xid 5c262126
[ 6695.660586] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff8801564e3800 xid e88fabdd

• Next a warning occurs, indicating a __list_add has detected a corrupted list, somewhere inside hash_delegation_locked

[ 7866.957210] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff880080868000 xid 84e07778
[ 7878.099182] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff8801394eb000 xid b31652cb
[ 7882.883142] ------------[ cut here ]------------
[ 7882.883156] WARNING: at lib/list_debug.c:29 __list_add+0x65/0xc0()
[ 7882.883157] list_add corruption. next->prev should be prev (ffff880230a5f068), but was ffff8801abc65868. (next=ffff8801abc65868).
[ 7882.883159] Modules linked in: nfsv3 nfs fscache binfmt_misc ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables ext4 mbcache jbd2 coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ppdev serio_raw vmw_balloon vmw_vmci i2c_piix4 parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_common vmwgfx
[ 7882.883231]  drm_kms_helper ttm ahci ata_piix libahci drm i2c_core vmxnet3 libata vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
[ 7882.883254] CPU: 3 PID: 2773 Comm: nfsd Not tainted 3.10.0-229.4.2.el7.x86_64 #1
[ 7882.883255] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[ 7882.883257]  ffff8800b5757bd8 00000000385971d4 ffff8800b5757b90 ffffffff816042d6
[ 7882.883259]  ffff8800b5757bc8 ffffffff8106e28b ffff880147a637c0 ffff8801abc65868
[ 7882.883261]  ffff880230a5f068 ffff8800b54443e0 ffff8801bee93a50 ffff8800b5757c30
[ 7882.883263] Call Trace:
[ 7882.883270]  [<ffffffff816042d6>] dump_stack+0x19/0x1b
[ 7882.883275]  [<ffffffff8106e28b>] warn_slowpath_common+0x6b/0xb0
[ 7882.883277]  [<ffffffff8106e32c>] warn_slowpath_fmt+0x5c/0x80
[ 7882.883281]  [<ffffffff812d623b>] ? idr_alloc_cyclic+0x2b/0x60
[ 7882.883283]  [<ffffffff812ed685>] __list_add+0x65/0xc0
[ 7882.883294]  [<ffffffffa033278a>] hash_delegation_locked+0x3a/0x40 [nfsd]
[ 7882.883300]  [<ffffffffa0338b29>] nfsd4_process_open2+0x979/0xfc0 [nfsd]
[ 7882.883306]  [<ffffffffa0327a8a>] nfsd4_open+0x55a/0x850 [nfsd]
[ 7882.883311]  [<ffffffffa0328257>] nfsd4_proc_compound+0x4d7/0x7f0 [nfsd]
[ 7882.883316]  [<ffffffffa0313e1b>] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 7882.883328]  [<ffffffffa02d9b33>] svc_process_common+0x453/0x6f0 [sunrpc]
[ 7882.883336]  [<ffffffffa02d9ed3>] svc_process+0x103/0x170 [sunrpc]
[ 7882.883340]  [<ffffffffa03137a7>] nfsd+0xe7/0x150 [nfsd]
[ 7882.883345]  [<ffffffffa03136c0>] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 7882.883347]  [<ffffffff8109726f>] kthread+0xcf/0xe0
[ 7882.883349]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[ 7882.883352]  [<ffffffff816140bc>] ret_from_fork+0x7c/0xb0
[ 7882.883353]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[ 7882.883355] ---[ end trace 2c830e49e095cf84 ]---

• finally, a kernel panic on NFS server, with RIP locks_in_grace

[ 7893.172876] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff880231a96000 xid 7cff4b0e
[ 7894.631260] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt ffff8801abc65000 xid f47a468f
[ 7894.799450] general protection fault: 0000 [#1] SMP
[ 7894.799519] Modules linked in: fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat nfsv3 nfs fscache binfmt_misc ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables ext4 mbcache jbd2 coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ppdev serio_raw vmw_balloon vmw_vmci i2c_piix4 parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sr_mod cdrom ata_generic
[ 7894.799778]  pata_acpi sd_mod crc_t10dif crct10dif_common vmwgfx drm_kms_helper ttm ahci ata_piix libahci drm i2c_core vmxnet3 libata vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
[ 7894.799843] CPU: 3 PID: 2763 Comm: nfsd Tainted: G        W   --------------   3.10.0-229.4.2.el7.x86_64 #1
[ 7894.799870] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
[ 7894.799899] task: ffff8800bb382d80 ti: ffff88023342c000 task.ti: ffff88023342c000
[ 7894.799920] RIP: 0010:[<ffffffffa0298620>]  [<ffffffffa0298620>] locks_in_grace+0x30/0x50 [lockd]
[ 7894.799953] RSP: 0018:ffff88023342fc78  EFLAGS: 00010202
[ 7894.799968] RAX: 002fffff0002002c RBX: ffff880182cb6000 RCX: 0000000000000001
[ 7894.799988] RDX: 002fffff000200d4 RSI: 0000000000000000 RDI: ffffea0004cea100
[ 7894.800008] RBP: ffff88023342fd28 R08: ffff880230a5f000 R09: 0000000000000000
[ 7894.800028] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 7894.800047] R13: 0000000000000001 R14: ffff8802315303e0 R15: ffff8801bee93a50
[ 7894.800071] FS:  0000000000000000(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
[ 7894.800093] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7894.800110] CR2: 00007f671823b7b0 CR3: 00000000bb199000 CR4: 00000000001407e0
[ 7894.800165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7894.800216] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7894.800237] Stack:
[ 7894.800245]  ffffffffa0338999 ffff880200000001 ffff880222d84a50 ffff880230a5f000
[ 7894.800270]  ffff880231531068 ffff880231531000 0000000000000008 0000000000000000
[ 7894.800294]  0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 7894.800318] Call Trace:
[ 7894.800350]  [<ffffffffa0338999>] ? nfsd4_process_open2+0x7e9/0xfc0 [nfsd]
[ 7894.800375]  [<ffffffffa0327a8a>] nfsd4_open+0x55a/0x850 [nfsd]
[ 7894.800396]  [<ffffffffa0328257>] nfsd4_proc_compound+0x4d7/0x7f0 [nfsd]
[ 7894.800419]  [<ffffffffa0313e1b>] nfsd_dispatch+0xbb/0x200 [nfsd]
[ 7894.800448]  [<ffffffffa02d9b33>] svc_process_common+0x453/0x6f0 [sunrpc]
[ 7894.800473]  [<ffffffffa02d9ed3>] svc_process+0x103/0x170 [sunrpc]
[ 7894.800494]  [<ffffffffa03137a7>] nfsd+0xe7/0x150 [nfsd]
[ 7894.800513]  [<ffffffffa03136c0>] ? nfsd_destroy+0x80/0x80 [nfsd]
[ 7894.800534]  [<ffffffff8109726f>] kthread+0xcf/0xe0
[ 7894.800550]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[ 7894.800571]  [<ffffffff816140bc>] ret_from_fork+0x7c/0xb0
[ 7894.800587]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[ 7894.800606] Code: 8b 05 45 87 00 00 85 c0 48 8b 97 e8 0c 00 00 74 28 3b 02 77 24 83 e8 01 48 98 48 8b 44 c2 18 48 85 c0 74 17 48 8d 90 a8 00 00 00 <48> 39 90 a8 00 00 00 0f 95 c0 0f b6 c0 c3 0f 0b 55 48 89 e5 e8
[ 7894.800710] RIP  [<ffffffffa0298620>] locks_in_grace+0x30/0x50 [lockd]
[ 7894.800735]  RSP <ffff88023342fc78>