RHEL 5 cluster node using GFS2 kernel panics in filemap_fdatawait, __filemap_fdatawrite_range, or __filemap_fdatawrite_range
Issue
- Cluster node with
GFS2filesystems mounted had a system panic:
Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
[<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
PGD affb1c067 PUD b01ff3067 PMD 0
Oops: 0000 [1] SMP
last sysfs file: /block/cciss!c0d0/cciss!c0d0p1/stat
CPU 7
Modules linked in: oracleacfs(PFU) oracleadvm(PFU) oracleoks(PU)
nfs nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 ipmi_devintf ipmi_si
ipmi_msghandler lock_dlm gfs2 dlm configfs sunrpc netxen_nic bonding ipv6 xfrm_nalgo crypto_api
emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh
video backlight sbs power_meter hwmon i2 c_ec i2c_core dell_wmi wmi button battery asus_acpi
acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom sg tpm_tis i7core_edac tpm
edac_mc tpm_bios hpilo serio_raw pcspkr shpchp e1000e dm_raid45 dm_message dm_region_hash
dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc ata_piix
libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 2658, comm: glock_workqueue Tainted: PF ---- 2.6.18-274.3.1.el5 #1
RIP: 0010:[<ffffffff800c8bb7>] [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
RSP: 0018:ffff811fb4e87db8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000009316d26 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff810511092778 R08: 0000000000000001 R09: ffff810511092778
R10: ffff81203f914008 R11: 0000000000000246 R12: 0000000000000000
R13: ffffffff888c5ee0 R14: ffff810511092700 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff81203fe003c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000b00e5b000 CR4: 00000000000006a0
Process glock_workqueue (pid: 2658, threadinfo ffff811fb4e86000, task ffff81202d058820)
Stack: ffffffff888af6d6 0000000300000000 ffff810511092778 ffff8105110927c0
ffffffff888ae125 ffff811fbe273000 ffff810511092778 0000000000000000
0000000000000000 0000000000000282 ffff810511092778 ffffffff888aeb4f
Call Trace:
[<ffffffff888af6d6>] :gfs2:inode_go_sync+0x117/0x125
[<ffffffff888ae125>] :gfs2:do_xmote+0xdd/0x1c1
[<ffffffff888aeb4f>] :gfs2:glock_work_func+0x0/0xe2
[<ffffffff888aebd4>] :gfs2:glock_work_func+0x85/0xe2
[<ffffffff8004d311>] run_workqueue+0x9e/0xfb
[<ffffffff80049b20>] worker_thread+0x0/0x122
[<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
[<ffffffff80049c10>] worker_thread+0xf0/0x122
[<ffffffff8008e7f7>] default_wake_function+0x0/0xe
[<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
[<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032722>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032624>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11
Code: 48 8b 07 48 8b 40 60 48 85 c0 74 0f 48 8d 50 ff 31 f6 48 c1
RIP [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
RSP <ffff811fb4e87db8>
CR2: 0000000000000000
<0>Kernel panic - not syncing: Fatal exception
- Cluster node with
GFS2filesystems mounted had a system panic:
Unable to handle kernel paging request at 000000021abf33c8 RIP:
[<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
PGD 881c49067 PUD 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:02:00.3/host2/rport-2:0-3/target2:0:0/2:0:0:6/timeout
CPU 17
Modules linked in: nfs nfs_acl mptctl mptbase ipmi_watchdog netconsole autofs4 ipmi_devintf ipmi_si ipmi_msghandler hidp rfcomm l2cap bluetooth lock_dlm gfs2 dlm configfs lockd sunrpc bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev be2net(U) tpm_tis shpchp i7core_edac sg serio_raw tpm hpilo tpm_bios pcspkr edac_mc 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 8708, comm: glock_workqueue Tainted: G ---- 2.6.18-348.el5 #1
RIP: 0010:[<ffffffff801596a8>] [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP: 0018:ffff8123f608dc58 EFLAGS: 00010082
RAX: 00000000534f2065 RBX: ffff810864775db8 RCX: 0000000000000015
RDX: 0000000000000000 RSI: ffff8123f608dd00 RDI: ffff810864775db8
RBP: 0000000000000000 R08: 0000000000000001 R09: ffffffff886bf340
R10: ffff810009049b10 R11: 0000000000000246 R12: 000000000000000e
R13: ffff810864775db0 R14: 000000000000000e R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff81120b8359c0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000021abf33c8 CR3: 0000000f8d615000 CR4: 00000000000006a0
Process glock_workqueue (pid: 8708, threadinfo ffff8123f608c000, task ffff811207d417e0)
Stack: 000000000000000e ffff8123f608dd00 0000000000000001 ffff8123f608dd00
000000000000000e ffff810864775db0 ffff8123f608dd78 0000000000000000
ffffffff8000ec43 ffff8123f608dcf0 ffff810864775db0 0000000000000000
Call Trace:
[<ffffffff8000ec43>] find_get_pages_tag+0x34/0x89
[<ffffffff80048286>] pagevec_lookup_tag+0x1a/0x21
[<ffffffff8004a2e7>] wait_on_page_writeback_range+0xdb/0x133
[<ffffffff8005aed3>] do_writepages+0x29/0x2f
[<ffffffff8004fd78>] __filemap_fdatawrite_range+0x50/0x5b
[<ffffffff886a87bc>] :gfs2:inode_go_sync+0x117/0x123
[<ffffffff886a71f6>] :gfs2:do_xmote+0xdd/0x1c1
[<ffffffff886a7b2c>] :gfs2:glock_work_func+0x0/0xe2
[<ffffffff886a7bb1>] :gfs2:glock_work_func+0x85/0xe2
[<ffffffff8004d8a1>] run_workqueue+0x9e/0xfb
[<ffffffff8004a0ea>] worker_thread+0x0/0x122
[<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
[<ffffffff8004a1da>] worker_thread+0xf0/0x122
[<ffffffff8008f34a>] default_wake_function+0x0/0xe
[<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032c26>] kthread+0xfe/0x132
[<ffffffff8005dfc1>] child_rip+0xa/0x11
[<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032b28>] kthread+0x0/0x132
[<ffffffff8005dfb7>] child_rip+0x0/0x11
Code: 4c 8b 3c c5 a0 30 46 80 b8 01 00 00 00 d3 e0 85 47 04 0f 84
RIP [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP <ffff8123f608dc58>
Environment
- Red Hat Enterprise Linux (RHEL) 5 with the Resilient Storage Add On
- GFS2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
