RHEL 6 Resilient Storage cluster node panics in unlock_buffer
Issue
- Cluster nodes are repeatedly panicking with the following log:
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff811a96e9>] unlock_buffer+0x9/0x20
PGD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:17.0/0000:13:00.0/irq
CPU 0
Modules linked in: gfs2 dlm configfs sunrpc vsock(U) be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ppdev parport_pc parport vmware_balloon vmxnet3 vmci(U) i2c_piix4 i2c_core sg shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 27358, comm: qmgr Not tainted 2.6.32-220.13.1.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffff811a96e9>] [<ffffffff811a96e9>] unlock_buffer+0x9/0x20
RSP: 0018:ffff880109d2ba78 EFLAGS: 00010296
RAX: ffff88010c36c668 RBX: ffff88010a1a37b0 RCX: ffff8800b9b12c48
RDX: 0000000000000000 RSI: ffff88010a1a37b0 RDI: 0000000000000000
RBP: ffff880109d2ba78 R08: 5010000000000000 R09: ff3837be4a130a02
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88010c36c668
R13: ffff88010c36c000 R14: ffff88010a1a3790 R15: ffff88003740a870
FS: 00007fb9cff477c0(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000010a6fc000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qmgr (pid: 27358, threadinfo ffff880109d2a000, task ffff88010a061580)
Stack:
ffff880109d2baa8 ffffffffa04035bf ffff8800a9c86c28 ffff88010c36c000
<0> ffff880109d2bb28 0000000000000000 ffff880109d2bac8 ffffffffa04192da
<0> 0000000000000003 ffff880134985418 ffff880109d2bd38 ffffffffa040dc7e
Call Trace:
[<ffffffffa04035bf>] buf_lo_add+0x6f/0x170 [gfs2]
[<ffffffffa04192da>] gfs2_trans_add_bh+0x3a/0x60 [gfs2]
[<ffffffffa040dc7e>] gfs2_rename+0x8ee/0x940 [gfs2]
[<ffffffffa040d4ca>] ? gfs2_rename+0x13a/0x940 [gfs2]
[<ffffffffa040d4e8>] ? gfs2_rename+0x158/0x940 [gfs2]
[<ffffffffa040d50c>] ? gfs2_rename+0x17c/0x940 [gfs2]
[<ffffffffa03fa51f>] ? gfs2_glock_add_to_lru+0x5f/0x90 [gfs2]
[<ffffffffa03fc043>] ? gfs2_holder_uninit+0x23/0x40 [gfs2]
[<ffffffffa0414ac0>] ? gfs2_rindex_hold+0x40/0xe0 [gfs2]
[<ffffffffa040d46a>] ? gfs2_rename+0xda/0x940 [gfs2]
[<ffffffff8118475b>] vfs_rename+0x3ab/0x440
[<ffffffff81186630>] sys_renameat+0x230/0x260
[<ffffffff8117b8f4>] ? cp_new_stat+0xe4/0x100
[<ffffffff8117bc06>] ? sys_newlstat+0x36/0x50
[<ffffffff8118667b>] sys_rename+0x1b/0x20
[<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Code: 48 8b 47 48 48 85 c0 74 04 31 f6 ff d0 e8 a0 3f 34 00 31 c0 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 <f0> 80 27 fb be 02 00 00 00 e8 f9 74 ee ff c9 c3 0f 1f 80 00 00
RIP [<ffffffff811a96e9>] unlock_buffer+0x9/0x20
RSP <ffff880109d2ba78>
CR2: 0000000000000000
Environment
- Red Hat Enterprise Linux (RHEL) 6 with the Resilient Storage Add on
- kernel prior to release 2.6.32-262.el6
- Mounted GFS2 file system(s)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
