kernel panic in __filemap_fdatawrite_range on a RHEL 6 Resilient Storage cluster node

Issue

• A cluster node running on Red Hat Enterprise Linux 6 with GFS2 had a panic with following sample back trace:

[785831.976947] general protection fault: 0000 [#1] SMP 
[785832.008156] last sysfs file: /sys/devices/system/cpu/cpu11/cpufreq/scaling_setspeed
[785832.055260] CPU 8 
[785832.065966] Modules linked in: iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables bluetooth rfkill gfs2 dlm drbd(U) mptctl mptbase configfs sunrpc pcc_cpufreq bonding ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log power_meter hpilo hpwdt bnx2 netxen_nic sg microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa(U) radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: nf_defrag_ipv4]
[785832.101003] 
[785832.101006] Modules linked in: iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables bluetooth rfkill gfs2 dlm drbd(U) mptctl mptbase configfs sunrpc pcc_cpufreq bonding ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log power_meter hpilo hpwdt bnx2 netxen_nic sg microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa(U) radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: nf_defrag_ipv4]
[785832.101029] Pid: 23855, comm: glock_workqueue Not tainted 2.6.32-131.12.1.el6.x86_64 #1 ProLiant DL360 G7
[785832.101031] RIP: 0010:[<ffffffff8110dade>]  [<ffffffff8110dade>] __filemap_fdatawrite_range+0x3e/0x60
[785832.101039] RSP: 0018:ffff8803c725bcc0  EFLAGS: 00010246
[785832.101040] RAX: 0000000000000000 RBX: ffff880301840b40 RCX: 7fffffffffffffff
[785832.101042] RDX: 7fffffffffffffff RSI: 0000000000000000 RDI: ffff8803c725bd28
[785832.101044] RBP: ffff8803c725bd30 R08: 0000f7edd53bfe80 R09: ffff8803c725bcc0
[785832.101045] R10: 0000000000000001 R11: 0000000000000001 R12: 0000f7edd53bfe80
[785832.101047] R13: ffff880301840ce0 R14: ffff880301840b90 R15: 0000000000000000
[785832.101049] FS:  0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[785832.101051] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[785832.101052] CR2: 00007f531d8fd028 CR3: 00000005d79e1000 CR4: 00000000000006e0
[785832.101054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[785832.101056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[785832.101058] Process glock_workqueue (pid: 23855, threadinfo ffff8803c725a000, task ffff88040996aa80)
[785832.101060] Stack:
[785832.101062]  0000000000000001 0000000000000000 0000000000000000 7fffffffffffffff
[785832.101066] <0> 0000000000000000 0000000000000000 7fffffffffffffff 0000000000000000
[785832.101070] <0> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[785832.101075] Call Trace:
[785832.101079]  [<ffffffff8110dfff>] filemap_fdatawrite+0x1f/0x30
[785832.101090]  [<ffffffffa05b5ea8>] inode_go_sync+0x98/0x160 [gfs2]
[785832.101098]  [<ffffffffa05b4b3b>] do_xmote+0x10b/0x250 [gfs2]
[785832.101106]  [<ffffffffa05b4d72>] run_queue+0xf2/0x170 [gfs2]
[785832.101114]  [<ffffffffa05b548b>] glock_work_func+0x7b/0x1b0 [gfs2]
[785832.101121]  [<ffffffffa05b5410>] ? glock_work_func+0x0/0x1b0 [gfs2]
[785832.101127]  [<ffffffff81088840>] worker_thread+0x170/0x2a0
[785832.101131]  [<ffffffff8108e180>] ? autoremove_wake_function+0x0/0x40
[785832.101134]  [<ffffffff810886d0>] ? worker_thread+0x0/0x2a0
[785832.101136]  [<ffffffff8108de16>] kthread+0x96/0xa0
[785832.101140]  [<ffffffff8100c1ca>] child_rip+0xa/0x20
[785832.101143]  [<ffffffff8108dd80>] ? kthread+0x0/0xa0
[785832.101145]  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
[785832.101147] Code: 49 89 f8 41 89 ca b9 0d 00 00 00 4c 89 cf f3 48 ab 48 b9 ff ff ff ff ff ff ff 7f 44 89 55 90 48 89 75 b8 48 89 4d a8 48 89 55 c0 <49> 8b 50 68 f6 42 30 02 74 08 c9 c3 66 0f 1f 44 00 00 4c 89 ce 
[785832.101162] RIP  [<ffffffff8110dade>] __filemap_fdatawrite_range+0x3e/0x60
[785832.101165]  RSP <ffff8803c725bcc0>
crash> 

• My cluster node kernel panicked with crash signature showing __filemap_fdatawrite_range and do_writepages:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffff811262f6>] do_writepages+0x16/0x40
PGD 3a5cdf067 PUD 24f8f6067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu4/cpufreq/scaling_setspeed
CPU 6 
Modules linked in: gfs2 dlm configfs pcc_cpufreq bonding ipv6 dm_round_robin scsi_dh_alua dm_multipath hpilo hpwdt sg microcode serio_raw be2iscsi iscsi_boot_sysfs libiscsi scsi_transport_iscsi iTCO_wdt iTCO_vendor_support be2net ioatdma dca power_meter shpchp ext4 mbcache jbd2 sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt hpsa video output dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs]

Pid: 14921, comm: glock_workqueue Tainted: G        W  ----------------   2.6.32-220.7.1.el6.x86_64 #1 HP ProLiant BL460c Gen8
RIP: 0010:[<ffffffff811262f6>]  [<ffffffff811262f6>] do_writepages+0x16/0x40
RSP: 0018:ffff880238787ca0  EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88024e461278 RCX: 7fffffffffffffff
RDX: ffff88032db4b658 RSI: ffff880238787cb0 RDI: ffff88032db4b5f0
RBP: ffff880238787ca0 R08: ffff88032db4b5f0 R09: ffff880238787cb0
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88032db4b5f0
R13: ffff88024e461410 R14: ffff88024e4612c0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880247440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 0000000395e09000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process glock_workqueue (pid: 14921, threadinfo ffff880238786000, task ffff8802268fd580)
Stack:
 ffff880238787d20 ffffffff8111121b 0000000000000001 0000000000000000
<0> 0000000000000000 7fffffffffffffff 0000000000000000 0000000000000000
<0> 7fffffffffffffff 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffff8111121b>] __filemap_fdatawrite_range+0x5b/0x60
 [<ffffffff8111171f>] filemap_fdatawrite+0x1f/0x30
 [<ffffffffa030072d>] inode_go_sync+0x9d/0x160 [gfs2]
 [<ffffffffa02ff2a9>] do_xmote+0x119/0x260 [gfs2]
 [<ffffffff814ecb0e>] ? thread_return+0x4e/0x760
 [<ffffffffa02ff4e1>] run_queue+0xf1/0x1d0 [gfs2]
 [<ffffffffa02ffc4a>] glock_work_func+0x7a/0x1b0 [gfs2]
 [<ffffffffa02ffbd0>] ? glock_work_func+0x0/0x1b0 [gfs2]
 [<ffffffff8108b150>] worker_thread+0x170/0x2a0
 [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8108afe0>] ? worker_thread+0x0/0x2a0
 [<ffffffff81090726>] kthread+0x96/0xa0
 [<ffffffff8100c14a>] child_rip+0xa/0x20
 [<ffffffff81090690>] ? kthread+0x0/0xa0
 [<ffffffff8100c140>] ? child_rip+0x0/0x20
Code: 30 4c 12 81 e8 3c fb ff ff c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 31 c0 48 83 7e 18 00 7e 0f 48 8b 47 58 <48> 8b 40 18 48 85 c0 74 09 ff d0 c9 c3 0f 1f 44 00 00 0f 1f 84 
RIP  [<ffffffff811262f6>] do_writepages+0x16/0x40
 RSP <ffff880238787ca0>
CR2: 0000000000000018