kernel panic in __filemap_fdatawrite_range on a RHEL 6 Resilient Storage cluster node
Issue
- A cluster node running on Red Hat Enterprise Linux 6 with GFS2 had a panic with following sample back trace:
[785831.976947] general protection fault: 0000 [#1] SMP
[785832.008156] last sysfs file: /sys/devices/system/cpu/cpu11/cpufreq/scaling_setspeed
[785832.055260] CPU 8
[785832.065966] Modules linked in: iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables bluetooth rfkill gfs2 dlm drbd(U) mptctl mptbase configfs sunrpc pcc_cpufreq bonding ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log power_meter hpilo hpwdt bnx2 netxen_nic sg microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa(U) radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: nf_defrag_ipv4]
[785832.101003]
[785832.101006] Modules linked in: iptable_filter iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_tables bluetooth rfkill gfs2 dlm drbd(U) mptctl mptbase configfs sunrpc pcc_cpufreq bonding ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log power_meter hpilo hpwdt bnx2 netxen_nic sg microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa(U) radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: nf_defrag_ipv4]
[785832.101029] Pid: 23855, comm: glock_workqueue Not tainted 2.6.32-131.12.1.el6.x86_64 #1 ProLiant DL360 G7
[785832.101031] RIP: 0010:[<ffffffff8110dade>] [<ffffffff8110dade>] __filemap_fdatawrite_range+0x3e/0x60
[785832.101039] RSP: 0018:ffff8803c725bcc0 EFLAGS: 00010246
[785832.101040] RAX: 0000000000000000 RBX: ffff880301840b40 RCX: 7fffffffffffffff
[785832.101042] RDX: 7fffffffffffffff RSI: 0000000000000000 RDI: ffff8803c725bd28
[785832.101044] RBP: ffff8803c725bd30 R08: 0000f7edd53bfe80 R09: ffff8803c725bcc0
[785832.101045] R10: 0000000000000001 R11: 0000000000000001 R12: 0000f7edd53bfe80
[785832.101047] R13: ffff880301840ce0 R14: ffff880301840b90 R15: 0000000000000000
[785832.101049] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
[785832.101051] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[785832.101052] CR2: 00007f531d8fd028 CR3: 00000005d79e1000 CR4: 00000000000006e0
[785832.101054] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[785832.101056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[785832.101058] Process glock_workqueue (pid: 23855, threadinfo ffff8803c725a000, task ffff88040996aa80)
[785832.101060] Stack:
[785832.101062] 0000000000000001 0000000000000000 0000000000000000 7fffffffffffffff
[785832.101066] <0> 0000000000000000 0000000000000000 7fffffffffffffff 0000000000000000
[785832.101070] <0> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[785832.101075] Call Trace:
[785832.101079] [<ffffffff8110dfff>] filemap_fdatawrite+0x1f/0x30
[785832.101090] [<ffffffffa05b5ea8>] inode_go_sync+0x98/0x160 [gfs2]
[785832.101098] [<ffffffffa05b4b3b>] do_xmote+0x10b/0x250 [gfs2]
[785832.101106] [<ffffffffa05b4d72>] run_queue+0xf2/0x170 [gfs2]
[785832.101114] [<ffffffffa05b548b>] glock_work_func+0x7b/0x1b0 [gfs2]
[785832.101121] [<ffffffffa05b5410>] ? glock_work_func+0x0/0x1b0 [gfs2]
[785832.101127] [<ffffffff81088840>] worker_thread+0x170/0x2a0
[785832.101131] [<ffffffff8108e180>] ? autoremove_wake_function+0x0/0x40
[785832.101134] [<ffffffff810886d0>] ? worker_thread+0x0/0x2a0
[785832.101136] [<ffffffff8108de16>] kthread+0x96/0xa0
[785832.101140] [<ffffffff8100c1ca>] child_rip+0xa/0x20
[785832.101143] [<ffffffff8108dd80>] ? kthread+0x0/0xa0
[785832.101145] [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
[785832.101147] Code: 49 89 f8 41 89 ca b9 0d 00 00 00 4c 89 cf f3 48 ab 48 b9 ff ff ff ff ff ff ff 7f 44 89 55 90 48 89 75 b8 48 89 4d a8 48 89 55 c0 <49> 8b 50 68 f6 42 30 02 74 08 c9 c3 66 0f 1f 44 00 00 4c 89 ce
[785832.101162] RIP [<ffffffff8110dade>] __filemap_fdatawrite_range+0x3e/0x60
[785832.101165] RSP <ffff8803c725bcc0>
crash>
- My cluster node kernel panicked with crash signature showing
__filemap_fdatawrite_rangeanddo_writepages:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
IP: [<ffffffff811262f6>] do_writepages+0x16/0x40
PGD 3a5cdf067 PUD 24f8f6067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu4/cpufreq/scaling_setspeed
CPU 6
Modules linked in: gfs2 dlm configfs pcc_cpufreq bonding ipv6 dm_round_robin scsi_dh_alua dm_multipath hpilo hpwdt sg microcode serio_raw be2iscsi iscsi_boot_sysfs libiscsi scsi_transport_iscsi iTCO_wdt iTCO_vendor_support be2net ioatdma dca power_meter shpchp ext4 mbcache jbd2 sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt hpsa video output dm_mirror dm_region_hash dm_log dm_mod [last unloaded: configfs]
Pid: 14921, comm: glock_workqueue Tainted: G W ---------------- 2.6.32-220.7.1.el6.x86_64 #1 HP ProLiant BL460c Gen8
RIP: 0010:[<ffffffff811262f6>] [<ffffffff811262f6>] do_writepages+0x16/0x40
RSP: 0018:ffff880238787ca0 EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff88024e461278 RCX: 7fffffffffffffff
RDX: ffff88032db4b658 RSI: ffff880238787cb0 RDI: ffff88032db4b5f0
RBP: ffff880238787ca0 R08: ffff88032db4b5f0 R09: ffff880238787cb0
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88032db4b5f0
R13: ffff88024e461410 R14: ffff88024e4612c0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff880247440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000018 CR3: 0000000395e09000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process glock_workqueue (pid: 14921, threadinfo ffff880238786000, task ffff8802268fd580)
Stack:
ffff880238787d20 ffffffff8111121b 0000000000000001 0000000000000000
<0> 0000000000000000 7fffffffffffffff 0000000000000000 0000000000000000
<0> 7fffffffffffffff 0000000000000000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff8111121b>] __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff8111171f>] filemap_fdatawrite+0x1f/0x30
[<ffffffffa030072d>] inode_go_sync+0x9d/0x160 [gfs2]
[<ffffffffa02ff2a9>] do_xmote+0x119/0x260 [gfs2]
[<ffffffff814ecb0e>] ? thread_return+0x4e/0x760
[<ffffffffa02ff4e1>] run_queue+0xf1/0x1d0 [gfs2]
[<ffffffffa02ffc4a>] glock_work_func+0x7a/0x1b0 [gfs2]
[<ffffffffa02ffbd0>] ? glock_work_func+0x0/0x1b0 [gfs2]
[<ffffffff8108b150>] worker_thread+0x170/0x2a0
[<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8108afe0>] ? worker_thread+0x0/0x2a0
[<ffffffff81090726>] kthread+0x96/0xa0
[<ffffffff8100c14a>] child_rip+0xa/0x20
[<ffffffff81090690>] ? kthread+0x0/0xa0
[<ffffffff8100c140>] ? child_rip+0x0/0x20
Code: 30 4c 12 81 e8 3c fb ff ff c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 31 c0 48 83 7e 18 00 7e 0f 48 8b 47 58 <48> 8b 40 18 48 85 c0 74 09 ff d0 c9 c3 0f 1f 44 00 00 0f 1f 84
RIP [<ffffffff811262f6>] do_writepages+0x16/0x40
RSP <ffff880238787ca0>
CR2: 0000000000000018
Environment
- Red Hat Enterprise Linux (RHEL) 6 with the Resilient Storage Add On
- kernel releases prior to
2.6.32-279.el6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
