RHEL 5 cluster node using GFS2 kernel panics in filemap_fdatawait, __filemap_fdatawrite_range, or __filemap_fdatawrite_range

Solution Unverified - Updated -

Issue

  • Cluster node with GFS2 filesystems mounted had a system panic:
 Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
 [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
  PGD affb1c067 PUD b01ff3067 PMD 0
  Oops: 0000 [1] SMP
  last sysfs file: /block/cciss!c0d0/cciss!c0d0p1/stat
  CPU 7
  Modules linked in: oracleacfs(PFU) oracleadvm(PFU) oracleoks(PU)
  nfs nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 ipmi_devintf ipmi_si
  ipmi_msghandler lock_dlm gfs2 dlm configfs sunrpc netxen_nic bonding ipv6 xfrm_nalgo crypto_api
  emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh
  video backlight sbs power_meter hwmon i2 c_ec i2c_core dell_wmi wmi button battery asus_acpi
  acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom sg tpm_tis i7core_edac tpm
  edac_mc tpm_bios hpilo serio_raw pcspkr shpchp e1000e dm_raid45 dm_message dm_region_hash
  dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc ata_piix
  libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  Pid: 2658, comm: glock_workqueue Tainted: PF    ---- 2.6.18-274.3.1.el5 #1
  RIP: 0010:[<ffffffff800c8bb7>] [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
  RSP: 0018:ffff811fb4e87db8  EFLAGS: 00010246
  RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
  RDX: 0000000009316d26 RSI: 0000000000000000 RDI: 0000000000000000
  RBP: ffff810511092778 R08: 0000000000000001 R09: ffff810511092778
  R10: ffff81203f914008 R11: 0000000000000246 R12: 0000000000000000
  R13: ffffffff888c5ee0 R14: ffff810511092700 R15: 0000000000000000
  FS:  0000000000000000(0000) GS:ffff81203fe003c0(0000) knlGS:0000000000000000
  CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
  CR2: 0000000000000000 CR3: 0000000b00e5b000 CR4: 00000000000006a0
  Process glock_workqueue (pid: 2658, threadinfo ffff811fb4e86000, task ffff81202d058820)
  Stack:  ffffffff888af6d6 0000000300000000 ffff810511092778 ffff8105110927c0
   ffffffff888ae125 ffff811fbe273000 ffff810511092778 0000000000000000
   0000000000000000 0000000000000282 ffff810511092778 ffffffff888aeb4f
  Call Trace:
   [<ffffffff888af6d6>] :gfs2:inode_go_sync+0x117/0x125
   [<ffffffff888ae125>] :gfs2:do_xmote+0xdd/0x1c1
   [<ffffffff888aeb4f>] :gfs2:glock_work_func+0x0/0xe2
   [<ffffffff888aebd4>] :gfs2:glock_work_func+0x85/0xe2
   [<ffffffff8004d311>] run_workqueue+0x9e/0xfb
   [<ffffffff80049b20>] worker_thread+0x0/0x122
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80049c10>] worker_thread+0xf0/0x122
   [<ffffffff8008e7f7>] default_wake_function+0x0/0xe
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80032722>] kthread+0xfe/0x132
   [<ffffffff8005dfb1>] child_rip+0xa/0x11
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80032624>] kthread+0x0/0x132
   [<ffffffff8005dfa7>] child_rip+0x0/0x11


  Code: 48 8b 07 48 8b 40 60 48 85 c0 74 0f 48 8d 50 ff 31 f6 48 c1
  RIP  [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
   RSP <ffff811fb4e87db8>
  CR2: 0000000000000000
   <0>Kernel panic - not syncing: Fatal exception
  • Cluster node with GFS2 filesystems mounted had a system panic:
Unable to handle kernel paging request at 000000021abf33c8 RIP: 
 [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
PGD 881c49067 PUD 0 
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:02:00.3/host2/rport-2:0-3/target2:0:0/2:0:0:6/timeout
CPU 17 
Modules linked in: nfs nfs_acl mptctl mptbase ipmi_watchdog netconsole autofs4 ipmi_devintf ipmi_si ipmi_msghandler hidp rfcomm l2cap bluetooth lock_dlm gfs2 dlm configfs lockd sunrpc bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev be2net(U) tpm_tis shpchp i7core_edac sg serio_raw tpm hpilo tpm_bios pcspkr edac_mc 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 8708, comm: glock_workqueue Tainted: G     ---- 2.6.18-348.el5 #1
RIP: 0010:[<ffffffff801596a8>]  [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP: 0018:ffff8123f608dc58  EFLAGS: 00010082
RAX: 00000000534f2065 RBX: ffff810864775db8 RCX: 0000000000000015
RDX: 0000000000000000 RSI: ffff8123f608dd00 RDI: ffff810864775db8
RBP: 0000000000000000 R08: 0000000000000001 R09: ffffffff886bf340
R10: ffff810009049b10 R11: 0000000000000246 R12: 000000000000000e
R13: ffff810864775db0 R14: 000000000000000e R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff81120b8359c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000021abf33c8 CR3: 0000000f8d615000 CR4: 00000000000006a0
Process glock_workqueue (pid: 8708, threadinfo ffff8123f608c000, task ffff811207d417e0)
Stack:  000000000000000e ffff8123f608dd00 0000000000000001 ffff8123f608dd00
 000000000000000e ffff810864775db0 ffff8123f608dd78 0000000000000000
 ffffffff8000ec43 ffff8123f608dcf0 ffff810864775db0 0000000000000000
Call Trace:
 [<ffffffff8000ec43>] find_get_pages_tag+0x34/0x89
 [<ffffffff80048286>] pagevec_lookup_tag+0x1a/0x21
 [<ffffffff8004a2e7>] wait_on_page_writeback_range+0xdb/0x133
 [<ffffffff8005aed3>] do_writepages+0x29/0x2f
 [<ffffffff8004fd78>] __filemap_fdatawrite_range+0x50/0x5b
 [<ffffffff886a87bc>] :gfs2:inode_go_sync+0x117/0x123
 [<ffffffff886a71f6>] :gfs2:do_xmote+0xdd/0x1c1
 [<ffffffff886a7b2c>] :gfs2:glock_work_func+0x0/0xe2
 [<ffffffff886a7bb1>] :gfs2:glock_work_func+0x85/0xe2
 [<ffffffff8004d8a1>] run_workqueue+0x9e/0xfb
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a1da>] worker_thread+0xf0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11


Code: 4c 8b 3c c5 a0 30 46 80 b8 01 00 00 00 d3 e0 85 47 04 0f 84 
RIP  [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP <ffff8123f608dc58>

Environment

  • Red Hat Enterprise Linux (RHEL) 5 with the Resilient Storage Add On
  • GFS2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.