RHEL6: NFS4 client kernel crash with RIP _spin_lock called from nfs4_locku_prepare .. nfs_wait_on_sequence due to corrupted nfs4_lock_state structure, size-512 slab corruption
Issue
- kernel crashed with the following message
[15919472.961774] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
[15919472.961979] IP: [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962096] PGD 5f27fff067 PUD 5f27ed3067 PMD 0
[15919472.962210] Oops: 0002 [#1] SMP
[15919472.962321] last sysfs file: /sys/devices/system/cpu/online
[15919472.962431] CPU 2
[15919472.962438] Modules linked in: bridge stp llc nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc iTCO_wdt iTCO_vendor_support microcode sg enic(U) power_meter sb_edac edac_core lpc_ich mfd_core shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif fnic(U) libfcoe libfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
[15919472.962533]
[15919472.962533] Pid: 3667, comm: rpciod/2 Tainted: G B --------------- 2.6.32-431.20.3.el6.x86_64 #1 Cisco Systems Inc UCSB-B200-M3/UCSB-B200-M3
[15919472.962533] RIP: 0010:[<ffffffff8152b21e>] [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962533] RSP: 0018:ffff883035e87d40 EFLAGS: 00010286
[15919472.962533] RAX: 0000000000010000 RBX: ffff8801cffc68c0 RCX: ffff883008ed1c00
[15919472.962533] RDX: 0000000000000081 RSI: ffff882f9ee47680 RDI: 00000000000000c8
[15919472.962533] RBP: ffff883035e87d40 R08: 0000000000000000 R09: 0000000000000000
[15919472.962533] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[15919472.962533] R13: 00000000000000c8 R14: ffff882f9ee47680 R15: ffffffffa01eb870
[15919472.962533] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[15919472.962533] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[15919472.962533] CR2: 00000000000000c8 CR3: 000000602e5f2000 CR4: 00000000000407e0
[15919472.962533] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[15919472.962533] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[15919472.962533] Process rpciod/2 (pid: 3667, threadinfo ffff883035e86000, task ffff883034244aa0)
[15919472.962533] Stack:
[15919472.962533] ffff883035e87d70 ffffffffa02f54ac ffff882da3128200 ffff882f9ee47680
[15919472.962533] <d> 0000000000000000 0000000000000001 ffff883035e87d90 ffffffffa02ea078
[15919472.962533] <d> ffff882f9ee47680 ffff882f9ee476f0 ffff883035e87da0 ffffffffa01eb883
[15919472.962533] Call Trace:
[15919472.962533] [<ffffffffa02f54ac>] nfs_wait_on_sequence+0x3c/0xb0 [nfs]
[15919472.962533] [<ffffffffa02ea078>] nfs4_locku_prepare+0x28/0xc0 [nfs]
[15919472.962533] [<ffffffffa01eb883>] rpc_prepare_task+0x13/0x20 [sunrpc]
[15919472.962533] [<ffffffffa01ed677>] __rpc_execute+0x77/0x350 [sunrpc]
[15919472.962533] [<ffffffffa01ed9f0>] ? rpc_async_schedule+0x0/0x40 [sunrpc]
[15919472.962533] [<ffffffffa01eda1a>] rpc_async_schedule+0x2a/0x40 [sunrpc]
[15919472.962533] [<ffffffff81094a20>] worker_thread+0x170/0x2a0
[15919472.962533] [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40
[15919472.962533] [<ffffffff810948b0>] ? worker_thread+0x0/0x2a0
[15919472.962533] [<ffffffff8109abf6>] kthread+0x96/0xa0
[15919472.962533] [<ffffffff8100c20a>] child_rip+0xa/0x20
[15919472.962533] [<ffffffff8109ab60>] ? kthread+0x0/0xa0
[15919472.962533] [<ffffffff8100c200>] ? child_rip+0x0/0x20
[15919472.962533] Code: e5 0f 1f 44 00 00 fa 66 0f 1f 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 82 3f d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 <f0> 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f b7 17 eb f5
[15919472.962533] RIP [<ffffffff8152b21e>] _spin_lock+0xe/0x30
[15919472.962533] RSP <ffff883035e87d40>
[15919472.962533] CR2: 00000000000000c8
Environment
- Red Hat Enterprise Linux 6
- seen on kernel-2.6.32-431.20.3.el6
- 3rd party modules
- fnic: Cisco FCoE HBA Driver, ver 1.6.0.12b
- enic: Cisco VIC Ethernet NIC Driver, ver 2.1.1.75
fnic (U)
enic (U)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
