RHEL6: kernel crash in NFS4 lock code path due to use-after-free of nfs4_lock_state
Issue
- NULL pointer dereference kernel crash similar to the following - at RIP
_raw_spin_lock
called fromnfs4_put_lock_state
BUG: unable to handle kernel NULL pointer dereference at 0000000000000134
IP: [<ffffffff8165270e>] _raw_spin_lock+0xe/0x30
<snip>
Call Trace:
[<ffffffff812f287d>] _atomic_dec_and_lock+0x4d/0x70
[<ffffffffa053c4f2>] nfs4_put_lock_state+0x32/0xb0 [nfsv4]
[<ffffffffa053c585>] nfs4_fl_release_lock+0x15/0x20 [nfsv4]
[<ffffffffa0522c06>] _nfs4_proc_getlk.isra.40+0x146/0x170 [nfsv4]
[<ffffffffa052ad99>] nfs4_proc_lock+0x399/0x5a0 [nfsv4]
- general protection fault kernel crash after NFS4 error message indicating
error 10008
which will cause retries, RIP of_spin_lock
called fromnfs_release_seqid
NFS: state manager: check lease failed on NFSv4 server nfs.example.com with error 10008
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 1
Modules linked in: edd nfs lockd fscache auth_rpcgss nfs_acl sunrpc vsock(U) ext3 jbd dm_multipath ppdev parport_pc parport microcode vmware_balloon vmci(U) i2c_piix4 i2c_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi e1000 pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod crc32c_intel be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]
Pid: 4859, comm: amqzxma0 Not tainted 2.6.32-504.39.1.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffff8152d6be>] [<ffffffff8152d6be>] _spin_lock+0xe/0x30
RSP: 0018:ffff880136d91b28 EFLAGS: 00010287
RAX: 0000000000010000 RBX: ffff880018a71580 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88013cca84a0 RDI: 0520a456500000c8
RBP: ffff880136d91b28 R08: ffff880136d90000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880018a71588
R13: 0520a45650000000 R14: 0520a456500000c8 R15: 00000000ffffd8e8
FS: 00007f8aa4339720(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0b904016a8 CR3: 000000013777e000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process amqzxma0 (pid: 4859, threadinfo ffff880136d90000, task ffff880138bfc040)
Stack:
ffff880136d91b58 ffffffffa04480a2 ffff880018a71580 0000000000000881
<d> ffff880139c2b200 00000000000000d0 ffff880136d91b78 ffffffffa04483c6
<d> ffff880136d91b98 ffff8800306f0600 ffff880136d91b98 ffffffffa0438389
Call Trace:
[<ffffffffa04480a2>] nfs_release_seqid+0x42/0x80 [nfs]
[<ffffffffa04483c6>] nfs_free_seqid+0x16/0x30 [nfs]
[<ffffffffa0438389>] nfs4_lock_release+0xa9/0x100 [nfs]
[<ffffffffa039db17>] rpc_release_calldata+0x17/0x20 [sunrpc]
[<ffffffffa039ef2e>] rpc_free_task+0x2e/0x70 [sunrpc]
[<ffffffffa039efc5>] rpc_final_put_task+0x55/0x60 [sunrpc]
[<ffffffffa039f000>] rpc_do_put_task+0x30/0x40 [sunrpc]
[<ffffffffa039f040>] rpc_put_task+0x10/0x20 [sunrpc]
[<ffffffffa0437afc>] _nfs4_do_setlk+0x35c/0x3f0 [nfs]
[<ffffffffa0437f45>] nfs4_proc_lock+0x3b5/0x500 [nfs]
[<ffffffffa0418b88>] do_setlk+0xf8/0x110 [nfs]
[<ffffffffa0418d67>] nfs_lock+0xd7/0x1d0 [nfs]
[<ffffffff811e0a63>] vfs_lock_file+0x23/0x40
[<ffffffff811e0cb7>] fcntl_setlk+0x177/0x320
[<ffffffff811a3457>] sys_fcntl+0x197/0x530
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
Code: e5 0f 1f 44 00 00 fa 66 0f 1f 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 b2 be d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 <f0> 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00
RIP [<ffffffff8152d6be>] _spin_lock+0xe/0x30
RSP <ffff880136d91b28>
Environment
- Red Hat Enterprise Linux 6
- kernel prior to kernel-2.6.32-696.el6 (errata kernel)
- believed to affect all kernels after 2.6.32-71.1.1.el6 (first kernel containing regression commit)
- seen on kernel-2.6.32-504.39.1.el6
- NFS4
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.