RHEL6: NFS4 client kernel crash with RIP _spin_lock called from nfs4_locku_prepare .. nfs_wait_on_sequence due to corrupted nfs4_lock_state structure, size-512 slab corruption

Issue

• kernel crashed with the following message

[15919472.961774] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
[15919472.961979] IP: [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962096] PGD 5f27fff067 PUD 5f27ed3067 PMD 0 
[15919472.962210] Oops: 0002 [#1] SMP 
[15919472.962321] last sysfs file: /sys/devices/system/cpu/online
[15919472.962431] CPU 2 
[15919472.962438] Modules linked in: bridge stp llc nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc iTCO_wdt iTCO_vendor_support microcode sg enic(U) power_meter sb_edac edac_core lpc_ich mfd_core shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif fnic(U) libfcoe libfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
[15919472.962533] 
[15919472.962533] Pid: 3667, comm: rpciod/2 Tainted: G    B      ---------------    2.6.32-431.20.3.el6.x86_64 #1 Cisco Systems Inc UCSB-B200-M3/UCSB-B200-M3
[15919472.962533] RIP: 0010:[<ffffffff8152b21e>]  [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962533] RSP: 0018:ffff883035e87d40  EFLAGS: 00010286
[15919472.962533] RAX: 0000000000010000 RBX: ffff8801cffc68c0 RCX: ffff883008ed1c00
[15919472.962533] RDX: 0000000000000081 RSI: ffff882f9ee47680 RDI: 00000000000000c8
[15919472.962533] RBP: ffff883035e87d40 R08: 0000000000000000 R09: 0000000000000000
[15919472.962533] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15919472.962533] R13: 00000000000000c8 R14: ffff882f9ee47680 R15: ffffffffa01eb870
[15919472.962533] FS:  0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[15919472.962533] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[15919472.962533] CR2: 00000000000000c8 CR3: 000000602e5f2000 CR4: 00000000000407e0
[15919472.962533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15919472.962533] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[15919472.962533] Process rpciod/2 (pid: 3667, threadinfo ffff883035e86000, task ffff883034244aa0)
[15919472.962533] Stack:
[15919472.962533]  ffff883035e87d70 ffffffffa02f54ac ffff882da3128200 ffff882f9ee47680
[15919472.962533] <d> 0000000000000000 0000000000000001 ffff883035e87d90 ffffffffa02ea078
[15919472.962533] <d> ffff882f9ee47680 ffff882f9ee476f0 ffff883035e87da0 ffffffffa01eb883
[15919472.962533] Call Trace:
[15919472.962533]  [<ffffffffa02f54ac>] nfs_wait_on_sequence+0x3c/0xb0 [nfs]
[15919472.962533]  [<ffffffffa02ea078>] nfs4_locku_prepare+0x28/0xc0 [nfs]
[15919472.962533]  [<ffffffffa01eb883>] rpc_prepare_task+0x13/0x20 [sunrpc]
[15919472.962533]  [<ffffffffa01ed677>] __rpc_execute+0x77/0x350 [sunrpc]
[15919472.962533]  [<ffffffffa01ed9f0>] ? rpc_async_schedule+0x0/0x40 [sunrpc]
[15919472.962533]  [<ffffffffa01eda1a>] rpc_async_schedule+0x2a/0x40 [sunrpc]
[15919472.962533]  [<ffffffff81094a20>] worker_thread+0x170/0x2a0
[15919472.962533]  [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[15919472.962533]  [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[15919472.962533]  [<ffffffff8109abf6>] kthread+0x96/0xa0
[15919472.962533]  [<ffffffff8100c20a>] child_rip+0xa/0x20
[15919472.962533]  [<ffffffff8109ab60>] ? kthread+0x0/0xa0
[15919472.962533]  [<ffffffff8100c200>] ? child_rip+0x0/0x20
[15919472.962533] Code: e5 0f 1f 44 00 00 fa 66 0f 1f 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 82 3f d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 <f0> 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 eb f5 
[15919472.962533] RIP  [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962533]  RSP <ffff883035e87d40>
[15919472.962533] CR2: 00000000000000c8