Kernel panics at fscache_put_operation+472 under heavy NFS load

Issue

• Kernel panics at fscache_put_operation+472 under heavy NFS load
• The following kernel log messages are observed at the time of the occurrence of the kernel panic.

[ 9610.374053] FS-Cache: Assertion failed
[ 9610.374103] 3 == 5 is false
[ 9610.374187] ------------[ cut here ]------------
[ 9610.374239] kernel BUG at fs/fscache/operation.c:419!
[ 9610.374290] invalid opcode: 0000 [#1] SMP 
[ 9610.374338] Modules linked in: fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 binfmt_misc sg rpcsec_gss_krb5 nfsv4 dns_resolver nfs ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables cachefiles fscache coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd serio_raw vmw_vmci pcspkr parport_pc parport i2c_piix4 shpchp mperf nfsd auth_rpcgss nfs_acl
[ 9610.375471]  lockd sunrpc uinput xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_common vmwgfx ttm drm ata_piix i2c_core vmxnet3 libata vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
[ 9610.375591] CPU: 2 PID: 25251 Comm: kworker/u16:4 Not tainted 3.10.0-123.4.4.el7.x86_64 #1
[ 9610.375626] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
[ 9610.375678] Workqueue: fscache_operation fscache_op_work_func [fscache]
[ 9610.375708] task: ffff8804274ba220 ti: ffff8801a5c30000 task.ti: ffff8801a5c30000
[ 9610.375739] RIP: 0010:[<ffffffffa03c6a48>]  [<ffffffffa03c6a48>] fscache_put_operation+0x1d8/0x210 [fscache]
[ 9610.375786] RSP: 0018:ffff8801a5c31de8  EFLAGS: 00010246
[ 9610.375810] RAX: 000000000000000f RBX: ffff880428778900 RCX: 0000000000000006
[ 9610.375840] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246
[ 9610.375870] RBP: ffff8801a5c31e00 R08: 0000000000000086 R09: 00000000000005a1
[ 9610.375900] R10: 0000000000000082 R11: 0000000000000000 R12: ffff88042365e580
[ 9610.375929] R13: ffff8804295b3800 R14: ffff880425078000 R15: 0000000000000200
[ 9610.375960] FS:  0000000000000000(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
[ 9610.377088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9610.377115] CR2: 00007f3b9e6cb000 CR3: 0000000425f20000 CR4: 00000000000407e0
[ 9610.377184] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9610.377249] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 9610.377280] Stack:
[ 9610.377293]  ffff880428778900 ffff88042365e580 ffff8804295b3800 ffff8801a5c31e18
[ 9610.377331]  ffffffffa03c7b0a ffff880428778900 ffff8801a5c31e60 ffffffff8107e03b
[ 9610.377368]  00000000295b3818 0000000000000000 ffff8804295b3818 ffff88042365e5b0
[ 9610.377403] Call Trace:
[ 9610.377428]  [<ffffffffa03c7b0a>] fscache_op_work_func+0x2a/0x50 [fscache]
[ 9610.377464]  [<ffffffff8107e03b>] process_one_work+0x17b/0x460
[ 9610.377493]  [<ffffffff8107ee0b>] worker_thread+0x11b/0x400
[ 9610.377520]  [<ffffffff8107ecf0>] ? rescuer_thread+0x400/0x400
[ 9610.377554]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[ 9610.377579]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[ 9610.377611]  [<ffffffff815f252c>] ret_from_fork+0x7c/0xb0
[ 9610.377905]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[ 9610.377935] Code: 31 c0 e8 03 4b 21 e1 48 c7 c7 e5 b3 3c a0 31 c0 e8 f5 4a 21 e1 8b 73 40 ba 05 00 00 00 48 c7 c7 30 b4 3c a0 31 c0 e8 df 4a 21 e1 <0f> 0b 48 c7 c7 e1 b3 3c a0 31 c0 e8 cf 4a 21 e1 48 c7 c7 e5 b3 
[ 9610.426984] RIP  [<ffffffffa03c6a48>] fscache_put_operation+0x1d8/0x210 [fscache]
[ 9610.428154]  RSP <ffff8801a5c31de8>