RHEL6: kernel crash in NFS4 lock code path due to use-after-free of nfs4_lock_state

Solution Unverified - Updated -

Issue

  • NULL pointer dereference kernel crash similar to the following - at RIP _raw_spin_lock called from nfs4_put_lock_state
 BUG: unable to handle kernel NULL pointer dereference at 0000000000000134
 IP: [<ffffffff8165270e>] _raw_spin_lock+0xe/0x30
<snip>
 Call Trace:
  [<ffffffff812f287d>] _atomic_dec_and_lock+0x4d/0x70
  [<ffffffffa053c4f2>] nfs4_put_lock_state+0x32/0xb0 [nfsv4]
  [<ffffffffa053c585>] nfs4_fl_release_lock+0x15/0x20 [nfsv4]
  [<ffffffffa0522c06>] _nfs4_proc_getlk.isra.40+0x146/0x170 [nfsv4]
  [<ffffffffa052ad99>] nfs4_proc_lock+0x399/0x5a0 [nfsv4]
  • general protection fault kernel crash after NFS4 error message indicating error 10008 which will cause retries, RIP of _spin_lock called from nfs_release_seqid
NFS: state manager: check lease failed on NFSv4 server nfs.example.com with error 10008
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 1
Modules linked in: edd nfs lockd fscache auth_rpcgss nfs_acl sunrpc vsock(U) ext3 jbd dm_multipath ppdev parport_pc parport microcode vmware_balloon vmci(U) i2c_piix4 i2c_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi e1000 pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod crc32c_intel be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]

Pid: 4859, comm: amqzxma0 Not tainted 2.6.32-504.39.1.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffff8152d6be>]  [<ffffffff8152d6be>] _spin_lock+0xe/0x30
RSP: 0018:ffff880136d91b28  EFLAGS: 00010287
RAX: 0000000000010000 RBX: ffff880018a71580 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88013cca84a0 RDI: 0520a456500000c8
RBP: ffff880136d91b28 R08: ffff880136d90000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880018a71588
R13: 0520a45650000000 R14: 0520a456500000c8 R15: 00000000ffffd8e8
FS:  00007f8aa4339720(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0b904016a8 CR3: 000000013777e000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process amqzxma0 (pid: 4859, threadinfo ffff880136d90000, task ffff880138bfc040)
Stack:
 ffff880136d91b58 ffffffffa04480a2 ffff880018a71580 0000000000000881
<d> ffff880139c2b200 00000000000000d0 ffff880136d91b78 ffffffffa04483c6
<d> ffff880136d91b98 ffff8800306f0600 ffff880136d91b98 ffffffffa0438389
Call Trace:
 [<ffffffffa04480a2>] nfs_release_seqid+0x42/0x80 [nfs]
 [<ffffffffa04483c6>] nfs_free_seqid+0x16/0x30 [nfs]
 [<ffffffffa0438389>] nfs4_lock_release+0xa9/0x100 [nfs]
 [<ffffffffa039db17>] rpc_release_calldata+0x17/0x20 [sunrpc]
 [<ffffffffa039ef2e>] rpc_free_task+0x2e/0x70 [sunrpc]
 [<ffffffffa039efc5>] rpc_final_put_task+0x55/0x60 [sunrpc]
 [<ffffffffa039f000>] rpc_do_put_task+0x30/0x40 [sunrpc]
 [<ffffffffa039f040>] rpc_put_task+0x10/0x20 [sunrpc]
 [<ffffffffa0437afc>] _nfs4_do_setlk+0x35c/0x3f0 [nfs]
 [<ffffffffa0437f45>] nfs4_proc_lock+0x3b5/0x500 [nfs]
 [<ffffffffa0418b88>] do_setlk+0xf8/0x110 [nfs]
 [<ffffffffa0418d67>] nfs_lock+0xd7/0x1d0 [nfs]
 [<ffffffff811e0a63>] vfs_lock_file+0x23/0x40
 [<ffffffff811e0cb7>] fcntl_setlk+0x177/0x320
 [<ffffffff811a3457>] sys_fcntl+0x197/0x530
 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
Code: e5 0f 1f 44 00 00 fa 66 0f 1f 44 00 00 f0 81 2f 00 00 00 01 74 05 e8 b2 be d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 <f0> 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 0f 1f 44 00 00
RIP  [<ffffffff8152d6be>] _spin_lock+0xe/0x30
 RSP <ffff880136d91b28>

Environment

  • Red Hat Enterprise Linux 6
    • kernel prior to kernel-2.6.32-696.el6 (errata kernel)
    • believed to affect all kernels after 2.6.32-71.1.1.el6 (first kernel containing regression commit)
    • seen on kernel-2.6.32-504.39.1.el6
  • NFS4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content