Issue

• Kernel panic in gfs2_quota_change
• Cluster node panics in gfs2_quota_hold
• My GFS2 node keeps crashing when I chown a file
• Node panics when the size of a file changes

BUG: unable to handle kernel NULL pointer dereference at 00000000000001a8
IP: [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
PGD 60f615067 PUD 6140d9067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host2/rport-2:0-1/target2:0:1/2:0:1:0/state
CPU 0 
Modules linked in: gfs2 nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 dlm configfs pcc_cpufreq 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support microcode serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core power_meter ses enclosure sg shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix bnx2 qla2xxx scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]

Pid: 9495, comm: nfsd Not tainted 2.6.32-431.el6.x86_64 #1 HP ProLiant DL380 G7
RIP: 0010:[<ffffffffa063c758>]  [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
RSP: 0018:ffff880613b47ba0  EFLAGS: 00010246
RAX: ffff8806103c6000 RBX: 00000000ffffffff RCX: 00000000000003fb
RDX: 0000000000000000 RSI: ffffffffffffffff RDI: ffff8804ac0e78e0
RBP: ffff880613b47bd0 R08: d010000000000000 R09: fb35f21b9bbbda02
R10: 0000000000000022 R11: 0000000000000000 R12: ffff8804ac0e78e0
R13: 00000000000003fb R14: ffffffffffffffff R15: 00000000000003f4
FS:  0000000000000000(0000) GS:ffff88003f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000001a8 CR3: 000000060d345000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process nfsd (pid: 9495, threadinfo ffff880613b46000, task ffff880613128080)
Stack:
 ffff880613a2f1c0 0000000000000000 ffff8804ac0e78e0 ffff880613b47c00
<d> 0000000000000001 00000000000003f4 ffff880613b47c80 ffffffffa0639c66
<d> 0100000000000040 ffffffff000003fb ffff8806103c6000 ffffffff13b47bf0
Call Trace:
 [<ffffffffa0639c66>] gfs2_setattr+0x2d6/0x320 [gfs2]
 [<ffffffffa06399d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffff811a7338>] notify_change+0x168/0x340
 [<ffffffff8118aed0>] ? __sb_start_write+0x80/0x120
 [<ffffffffa05d280c>] nfsd_setattr+0x32c/0x3e0 [nfsd]
 [<ffffffffa05dffb4>] nfsd4_setattr+0x124/0x130 [nfsd]
 [<ffffffffa05e0458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
 [<ffffffffa05cd425>] nfsd_dispatch+0xe5/0x230 [nfsd]
 [<ffffffffa05587e4>] svc_process_common+0x344/0x640 [sunrpc]
 [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
 [<ffffffffa0558e20>] svc_process+0x110/0x160 [sunrpc]
 [<ffffffffa05cdb52>] nfsd+0xc2/0x160 [nfsd]
 [<ffffffffa05cda90>] ? nfsd+0x0/0x160 [nfsd]
 [<ffffffff8109aef6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ae60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
Code: 1f 44 00 00 48 85 f6 49 89 fc 49 89 f6 89 d3 41 89 cd 0f 84 83 00 00 00 41 f6 84 24 65 03 00 00 02 75 3b 49 8b 94 24 00 03 00 00 <44> 8b ba a8 01 00 00 45 85 ff 74 27 45 31 ff 44 89 f8 48 8b bc 
RIP  [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
 RSP <ffff880613b47ba0>
CR2: 00000000000001a8

BUG: unable to handle kernel NULL pointer dereference at 00000000000001a8
IP: [<ffffffffa05cdea4>] gfs2_quota_hold+0xd4/0x200 [gfs2]
PGD 2f927b067 PUD 2f94fa067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:05.0/0000:1f:00.1/host5/rport-5:0-1/fc_remote_ports/rport-5:0-1/roles
CPU 6 
Modules linked in: gfs2 dlm configfs iptable_filter ip_tables ebtable_nat ebtables nfs lockd fscache auth_rpcgss nfs_acl bnx2fc cnic uio fcoe libfcoe libfc 8021q garp sunrpc ipmi_devintf acpi_cpufreq freq_table mperf bridge stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 vfat fat dm_round_robin vhost_net macvtap macvlan tun kvm_intel kvm microcode iTCO_wdt iTCO_vendor_support e1000e ptp pps_core serio_raw sg i2c_i801 i2c_core lpc_ich mfd_core cdc_ether usbnet mii shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptsas mptscsih mptbase scsi_transport_sas qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic ata_piix dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 22994, comm: chown Not tainted 2.6.32-431.el6.x86_64 #1 IBM System x3250 M3 -[4251AC1]-/69Y5276     
RIP: 0010:[<ffffffffa05cdea4>]  [<ffffffffa05cdea4>] gfs2_quota_hold+0xd4/0x200 [gfs2]
RSP: 0018:ffff8802b9bafd08  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8802d4f9b860 RCX: ffff8802f434a298
RDX: ffff8802f9b6f018 RSI: 0000000000000000 RDI: ffff8802f9241000
RBP: ffff8802b9bafd48 R08: f010000000000000 R09: fd0d09eec5a99e02
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8802f434a208
R13: ffff8802f9241000 R14: 00000000000001f7 R15: 00000000000001f8
FS:  00007fc7518fe700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001a8 CR3: 00000002b99c6000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process chown (pid: 22994, threadinfo ffff8802b9bae000, task ffff8802f8ef4040)
Stack:
 ffff8802b9bafd28 ffffffffa05b9025 ffff8802b9bafdb8 ffff8802d4f9b860
<d> ffff8802f9241000 ffff8802b9bafdb8 ffff8802b9bafeb8 00000000000001f8
<d> ffff8802b9bafd88 ffffffffa05ce35f ffff8802b9bafd88 ffffffff811a719f
Call Trace:
 [<ffffffffa05b9025>] ? gfs2_glock_wait+0x25/0x90 [gfs2]
 [<ffffffffa05ce35f>] gfs2_quota_lock+0x2f/0x160 [gfs2]
 [<ffffffff811a719f>] ? inode_change_ok+0x19f/0x1d0
 [<ffffffffa05cab24>] gfs2_setattr+0x194/0x320 [gfs2]
 [<ffffffffa05ca9d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffff811a7338>] notify_change+0x168/0x340
 [<ffffffff811863fe>] chown_common+0x6e/0x90
 [<ffffffff811866cf>] sys_fchownat+0xbf/0xe0
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: c0 74 14 48 89 df 89 45 c8 e8 f9 f5 ff ff 8b 45 c8 eb b7 0f 1f 40 00 48 8b 83 00 03 00 00 49 8d 8c 24 90 00 00 00 31 f6 4c 89 ef <83> 80 a8 01 00 00 01 8b 53 54 e8 9d fd ff ff 85 c0 75 c5 48 8b 
RIP  [<ffffffffa05cdea4>] gfs2_quota_hold+0xd4/0x200 [gfs2]

BUG: unable to handle kernel NULL pointer dereference at 00000000000001a8
IP: [<ffffffffa05d2758>] gfs2_quota_change+0x48/0xf0 [gfs2]
PGD 2ea5e9067 PUD 2f9279067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:05.0/0000:1f:00.1/host1/rport-1:0-1/fc_remote_ports/rport-1:0-1/roles
CPU 6 
Modules linked in: gfs2 dlm configfs iptable_filter ip_tables ebtable_nat ebtables nfs lockd fscache auth_rpcgss nfs_acl bnx2fc cnic uio fcoe libfcoe libfc 8021q garp sunrpc ipmi_devintf acpi_cpufreq freq_table mperf bridge stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 vfat fat dm_round_robin vhost_net macvtap macvlan tun kvm_intel kvm microcode iTCO_wdt iTCO_vendor_support e1000e ptp pps_core serio_raw sg i2c_i801 i2c_core lpc_ich mfd_core cdc_ether usbnet mii shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif mptsas mptscsih mptbase scsi_transport_sas pata_acpi ata_generic ata_piix qla2xxx scsi_transport_fc scsi_tgt dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 12383, comm: chown Not tainted 2.6.32-431.el6.x86_64 #1 IBM System x3250 M3 -[4251AC1]-/69Y5276     
RIP: 0010:[<ffffffffa05d2758>]  [<ffffffffa05d2758>] gfs2_quota_change+0x48/0xf0 [gfs2]
RSP: 0018:ffff8802d6477d58  EFLAGS: 00010246
RAX: ffff8802f9b47000 RBX: 00000000000001f6 RCX: 00000000000001f7
RDX: 0000000000000000 RSI: fffffffffffffffe RDI: ffff8802db4b8470
RBP: ffff8802d6477d88 R08: d010000000000000 R09: fd06a6fdee663a02
R10: 0000000000000010 R11: 0000000000000000 R12: ffff8802db4b8470
R13: 00000000000001f7 R14: fffffffffffffffe R15: 00000000000001f8
FS:  00007f21a869f700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001a8 CR3: 00000002d7ffd000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process chown (pid: 12383, threadinfo ffff8802d6476000, task ffff8802f9b61540)
Stack:
 ffff8802d6477eb8 0000000000000000 ffff8802db4b8470 ffff8802d6477db8
<d> 0000000000000002 00000000000001f8 ffff8802d6477e38 ffffffffa05cfc66
<d> 0100000000000000 000001f6000001f7 ffff8802f9b47000 000001f7d6477da8
Call Trace:
 [<ffffffffa05cfc66>] gfs2_setattr+0x2d6/0x320 [gfs2]
 [<ffffffffa05cf9d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffff811a7338>] notify_change+0x168/0x340
 [<ffffffff811863fe>] chown_common+0x6e/0x90
 [<ffffffff811866cf>] sys_fchownat+0xbf/0xe0
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 1f 44 00 00 48 85 f6 49 89 fc 49 89 f6 89 d3 41 89 cd 0f 84 83 00 00 00 41 f6 84 24 65 03 00 00 02 75 3b 49 8b 94 24 00 03 00 00 <44> 8b ba a8 01 00 00 45 85 ff 74 27 45 31 ff 44 89 f8 48 8b bc 
RIP  [<ffffffffa05d2758>] gfs2_quota_change+0x48/0xf0 [gfs2]

BUG: unable to handle kernel NULL pointer dereference at 00000000000001a8
IP: [<ffffffffa03a4318>] gfs2_quota_change+0x48/0xf0 [gfs2]
PGD caa5067 PUD 3d9e2067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/virtio0/net/eth0/broadcast
CPU 0
Modules linked in: gfs2 dlm configfs sctp libcrc32c nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc ipv6 microcode virtio_balloon
virtio_net i2c_piix4 i2c_core ext4 mbcache jbd2 virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 4512, comm: genesis Not tainted 2.6.32-338.el6.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffffa03a4318>]  [<ffffffffa03a4318>] gfs2_quota_change+0x48/0xf0 [gfs2]
RSP: 0018:ffff88000d477af8  EFLAGS: 00010246
RAX: 0000000000000003 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: fffffffffffffffd RDI: ffff88003b04a880
RBP: ffff88000d477b28 R08: 0010000000000000 R09: ffa4f2a6c3522002
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88003b04a880
R13: 0000000000000000 R14: fffffffffffffffd R15: 0000000000000000
FS:  00007f78c6cb5700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000001a8 CR3: 0000000021ae7000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process genesis (pid: 4512, threadinfo ffff88000d476000, task ffff880006164080)
Stack:
 0000000000000001 ffff88003b04a880 ffff88000f3438e8 0000000000000000
<d> 0000000000000001 0000000000000000 ffff88000d477bf8 ffffffffa03831c5
<d> ffff88000d477b88 ffff88003b04a9a8 ffff880000000003 ffff880000000000
Call Trace:
 [<ffffffffa03831c5>] recursive_scan+0x695/0x750 [gfs2]
 [<ffffffffa0383365>] trunc_dealloc+0xe5/0x100 [gfs2]
 [<ffffffffa039039f>] ? gfs2_glock_put+0x3f/0x180 [gfs2]
 [<ffffffff81182b6d>] ? __sb_end_write+0x3d/0x70
 [<ffffffffa0384abb>] do_shrink+0x9b/0x4b0 [gfs2]
 [<ffffffffa0392208>] ? do_promote+0x208/0x330 [gfs2]
 [<ffffffffa0384f5e>] gfs2_setattr_size+0x5e/0x210 [gfs2]
 [<ffffffffa03a1524>] gfs2_setattr+0x214/0x330 [gfs2]
 [<ffffffffa03a1356>] ? gfs2_setattr+0x46/0x330 [gfs2]
 [<ffffffff8119e268>] notify_change+0x168/0x340
 [<ffffffff8117ee74>] do_truncate+0x64/0xa0
 [<ffffffff8117f170>] sys_ftruncate+0x120/0x130
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 1f 44 00 00 48 85 f6 49 89 fc 49 89 f6 89 d3 41 89 cd 0f 84 83 00 00 00 41 f6 84 24 55 03 00 00 02 75 3b 49 8b 94 24 00 03 00 00
<44> 8b 9a a8 01 00 00 45 85 db 74 27 45 31 ff 44 89 f8 48 8b bc
RIP  [<ffffffffa03a4318>] gfs2_quota_change+0x48/0xf0 [gfs2]
 RSP <ffff88000d477af8>
CR2: 00000000000001a8

BUG: unable to handle kernel NULL pointer dereference at 00000000000001a8
IP: [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
PGD 60f615067 PUD 6140d9067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host2/rport-2:0-1/target2:0:1/2:0:1:0/state
CPU 0 
Modules linked in: gfs2 nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 dlm configfs pcc_cpufreq 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support microcode serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core power_meter ses enclosure sg shpchp ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix bnx2 qla2xxx scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi [last unloaded: scsi_wait_scan]

Pid: 9495, comm: nfsd Not tainted 2.6.32-431.el6.x86_64 #1 HP ProLiant DL380 G7
RIP: 0010:[<ffffffffa063c758>]  [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
RSP: 0018:ffff880613b47ba0  EFLAGS: 00010246
RAX: ffff8806103c6000 RBX: 00000000ffffffff RCX: 00000000000003fb
RDX: 0000000000000000 RSI: ffffffffffffffff RDI: ffff8804ac0e78e0
RBP: ffff880613b47bd0 R08: d010000000000000 R09: fb35f21b9bbbda02
R10: 0000000000000022 R11: 0000000000000000 R12: ffff8804ac0e78e0
R13: 00000000000003fb R14: ffffffffffffffff R15: 00000000000003f4
FS:  0000000000000000(0000) GS:ffff88003f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000000001a8 CR3: 000000060d345000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process nfsd (pid: 9495, threadinfo ffff880613b46000, task ffff880613128080)
Stack:
 ffff880613a2f1c0 0000000000000000 ffff8804ac0e78e0 ffff880613b47c00
<d> 0000000000000001 00000000000003f4 ffff880613b47c80 ffffffffa0639c66
<d> 0100000000000040 ffffffff000003fb ffff8806103c6000 ffffffff13b47bf0
Call Trace:
 [<ffffffffa0639c66>] gfs2_setattr+0x2d6/0x320 [gfs2]
 [<ffffffffa06399d6>] ? gfs2_setattr+0x46/0x320 [gfs2]
 [<ffffffff811a7338>] notify_change+0x168/0x340
 [<ffffffff8118aed0>] ? __sb_start_write+0x80/0x120
 [<ffffffffa05d280c>] nfsd_setattr+0x32c/0x3e0 [nfsd]
 [<ffffffffa05dffb4>] nfsd4_setattr+0x124/0x130 [nfsd]
 [<ffffffffa05e0458>] nfsd4_proc_compound+0x3d8/0x490 [nfsd]
 [<ffffffffa05cd425>] nfsd_dispatch+0xe5/0x230 [nfsd]
 [<ffffffffa05587e4>] svc_process_common+0x344/0x640 [sunrpc]
 [<ffffffff81065df0>] ? default_wake_function+0x0/0x20
 [<ffffffffa0558e20>] svc_process+0x110/0x160 [sunrpc]
 [<ffffffffa05cdb52>] nfsd+0xc2/0x160 [nfsd]
 [<ffffffffa05cda90>] ? nfsd+0x0/0x160 [nfsd]
 [<ffffffff8109aef6>] kthread+0x96/0xa0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109ae60>] ? kthread+0x0/0xa0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20
Code: 1f 44 00 00 48 85 f6 49 89 fc 49 89 f6 89 d3 41 89 cd 0f 84 83 00 00 00 41 f6 84 24 65 03 00 00 02 75 3b 49 8b 94 24 00 03 00 00 <44> 8b ba a8 01 00 00 45 85 ff 74 27 45 31 ff 44 89 f8 48 8b bc 
RIP  [<ffffffffa063c758>] gfs2_quota_change+0x48/0xf0 [gfs2]
 RSP <ffff880613b47ba0>
CR2: 00000000000001a8