RHEL 5 cluster node using GFS2 kernel panics in filemap_fdatawait, __filemap_fdatawrite_range, or __filemap_fdatawrite_range

Issue

• Cluster node with GFS2 filesystems mounted had a system panic:

 Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
 [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
  PGD affb1c067 PUD b01ff3067 PMD 0
  Oops: 0000 [1] SMP
  last sysfs file: /block/cciss!c0d0/cciss!c0d0p1/stat
  CPU 7
  Modules linked in: oracleacfs(PFU) oracleadvm(PFU) oracleoks(PU)
  nfs nfsd exportfs lockd nfs_acl auth_rpcgss oracleasm(U) autofs4 ipmi_devintf ipmi_si
  ipmi_msghandler lock_dlm gfs2 dlm configfs sunrpc netxen_nic bonding ipv6 xfrm_nalgo crypto_api
  emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh
  video backlight sbs power_meter hwmon i2 c_ec i2c_core dell_wmi wmi button battery asus_acpi
  acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom sg tpm_tis i7core_edac tpm
  edac_mc tpm_bios hpilo serio_raw pcspkr shpchp e1000e dm_raid45 dm_message dm_region_hash
  dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod qla2xxx scsi_transport_fc ata_piix
  libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
  Pid: 2658, comm: glock_workqueue Tainted: PF    ---- 2.6.18-274.3.1.el5 #1
  RIP: 0010:[<ffffffff800c8bb7>] [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
  RSP: 0018:ffff811fb4e87db8  EFLAGS: 00010246
  RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
  RDX: 0000000009316d26 RSI: 0000000000000000 RDI: 0000000000000000
  RBP: ffff810511092778 R08: 0000000000000001 R09: ffff810511092778
  R10: ffff81203f914008 R11: 0000000000000246 R12: 0000000000000000
  R13: ffffffff888c5ee0 R14: ffff810511092700 R15: 0000000000000000
  FS:  0000000000000000(0000) GS:ffff81203fe003c0(0000) knlGS:0000000000000000
  CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
  CR2: 0000000000000000 CR3: 0000000b00e5b000 CR4: 00000000000006a0
  Process glock_workqueue (pid: 2658, threadinfo ffff811fb4e86000, task ffff81202d058820)
  Stack:  ffffffff888af6d6 0000000300000000 ffff810511092778 ffff8105110927c0
   ffffffff888ae125 ffff811fbe273000 ffff810511092778 0000000000000000
   0000000000000000 0000000000000282 ffff810511092778 ffffffff888aeb4f
  Call Trace:
   [<ffffffff888af6d6>] :gfs2:inode_go_sync+0x117/0x125
   [<ffffffff888ae125>] :gfs2:do_xmote+0xdd/0x1c1
   [<ffffffff888aeb4f>] :gfs2:glock_work_func+0x0/0xe2
   [<ffffffff888aebd4>] :gfs2:glock_work_func+0x85/0xe2
   [<ffffffff8004d311>] run_workqueue+0x9e/0xfb
   [<ffffffff80049b20>] worker_thread+0x0/0x122
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80049c10>] worker_thread+0xf0/0x122
   [<ffffffff8008e7f7>] default_wake_function+0x0/0xe
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80032722>] kthread+0xfe/0x132
   [<ffffffff8005dfb1>] child_rip+0xa/0x11
   [<ffffffff800a2be5>] keventd_create_kthread+0x0/0xc4
   [<ffffffff80032624>] kthread+0x0/0x132
   [<ffffffff8005dfa7>] child_rip+0x0/0x11


  Code: 48 8b 07 48 8b 40 60 48 85 c0 74 0f 48 8d 50 ff 31 f6 48 c1
  RIP  [<ffffffff800c8bb7>] filemap_fdatawait+0x0/0x1e
   RSP <ffff811fb4e87db8>
  CR2: 0000000000000000
   <0>Kernel panic - not syncing: Fatal exception

• Cluster node with GFS2 filesystems mounted had a system panic:

Unable to handle kernel paging request at 000000021abf33c8 RIP: 
 [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
PGD 881c49067 PUD 0 
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:03.0/0000:02:00.3/host2/rport-2:0-3/target2:0:0/2:0:0:6/timeout
CPU 17 
Modules linked in: nfs nfs_acl mptctl mptbase ipmi_watchdog netconsole autofs4 ipmi_devintf ipmi_si ipmi_msghandler hidp rfcomm l2cap bluetooth lock_dlm gfs2 dlm configfs lockd sunrpc bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev be2net(U) tpm_tis shpchp i7core_edac sg serio_raw tpm hpilo tpm_bios pcspkr edac_mc 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod lpfc scsi_transport_fc cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 8708, comm: glock_workqueue Tainted: G     ---- 2.6.18-348.el5 #1
RIP: 0010:[<ffffffff801596a8>]  [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP: 0018:ffff8123f608dc58  EFLAGS: 00010082
RAX: 00000000534f2065 RBX: ffff810864775db8 RCX: 0000000000000015
RDX: 0000000000000000 RSI: ffff8123f608dd00 RDI: ffff810864775db8
RBP: 0000000000000000 R08: 0000000000000001 R09: ffffffff886bf340
R10: ffff810009049b10 R11: 0000000000000246 R12: 000000000000000e
R13: ffff810864775db0 R14: 000000000000000e R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff81120b8359c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000021abf33c8 CR3: 0000000f8d615000 CR4: 00000000000006a0
Process glock_workqueue (pid: 8708, threadinfo ffff8123f608c000, task ffff811207d417e0)
Stack:  000000000000000e ffff8123f608dd00 0000000000000001 ffff8123f608dd00
 000000000000000e ffff810864775db0 ffff8123f608dd78 0000000000000000
 ffffffff8000ec43 ffff8123f608dcf0 ffff810864775db0 0000000000000000
Call Trace:
 [<ffffffff8000ec43>] find_get_pages_tag+0x34/0x89
 [<ffffffff80048286>] pagevec_lookup_tag+0x1a/0x21
 [<ffffffff8004a2e7>] wait_on_page_writeback_range+0xdb/0x133
 [<ffffffff8005aed3>] do_writepages+0x29/0x2f
 [<ffffffff8004fd78>] __filemap_fdatawrite_range+0x50/0x5b
 [<ffffffff886a87bc>] :gfs2:inode_go_sync+0x117/0x123
 [<ffffffff886a71f6>] :gfs2:do_xmote+0xdd/0x1c1
 [<ffffffff886a7b2c>] :gfs2:glock_work_func+0x0/0xe2
 [<ffffffff886a7bb1>] :gfs2:glock_work_func+0x85/0xe2
 [<ffffffff8004d8a1>] run_workqueue+0x9e/0xfb
 [<ffffffff8004a0ea>] worker_thread+0x0/0x122
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8004a1da>] worker_thread+0xf0/0x122
 [<ffffffff8008f34a>] default_wake_function+0x0/0xe
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032c26>] kthread+0xfe/0x132
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff800a39aa>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032b28>] kthread+0x0/0x132
 [<ffffffff8005dfb7>] child_rip+0x0/0x11


Code: 4c 8b 3c c5 a0 30 46 80 b8 01 00 00 00 d3 e0 85 47 04 0f 84 
RIP  [<ffffffff801596a8>] radix_tree_gang_lookup_tag+0x21/0x19d
RSP <ffff8123f608dc58>