A gfs2 filesystem process `delete_workqueue` caused a kernel panic `BUG: unable to handle kernel NULL pointer dereference`

Solution In Progress - Updated -

Issue

  • A gfs2 filesystem process delete_workqueue caused a kernel panic BUG: unable to handle kernel NULL pointer dereference:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
IP: [<ffffffffa07e28a3>] gfs2_check_blk_type+0xb3/0x130 [gfs2]
PGD 33c9ab067 PUD 3a66b7067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 12 
Modules linked in: dcdbas mptctl mptbase autofs4 gfs2 dlm sctp libcrc32c configfs bonding ipv6 emcpdm(P)(U) emcpgpx(P)(U) emcpmpx(P)(U) emcp(P)(U) ipmi_devintf microcode power_meter acpi_ipmi ipmi_si ipmi_msghandler lpfc scsi_transport_fc scsi_tgt hpilo hpwdt bnx2 e1000e ptp pps_core serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom hpsa ata_generic pata_acpi pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]

Pid: 8747, comm: delete_workqueu Tainted: P        W  ---------------    2.6.32-504.30.3.el6.x86_64 #1 HP ProLiant DL385 G7
RIP: 0010:[<ffffffffa07e28a3>]  [<ffffffffa07e28a3>] gfs2_check_blk_type+0xb3/0x130 [gfs2]
RSP: 0018:ffff88043b047c60  EFLAGS: 00010202
RAX: 0000000000000018 RBX: 0000000000000000 RCX: 0000000000000004
RDX: 00000000000000c1 RSI: 0000000000000306 RDI: ffff88043b047c60
RBP: ffff88043b047cf0 R08: 000000000000fe80 R09: 00000000ffffffff
R10: 0021c64d00fc0b86 R11: 0000000000000001 R12: ffff88043b047c60
R13: 0000000008cf7644 R14: 0000000000000002 R15: 0000000000000000
FS:  00007faaeb3ff700(0000) GS:ffff880247500000(0000) knlGS:00000000f5f06b70
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 00000003526ab000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process delete_workqueu (pid: 8747, threadinfo ffff88043b046000, task ffff88043bea5520)
Stack:
 ffff88005ebeeab8 ffff88005ebeeab8 ffff88005ebeea68 ffff88043b6e0a40
<d> 0000000000000003 ffff880200000000 0000000000000040 ffffffffa07e2850
<d> ffff8801e62a5ca8 ffff88004f3de920 0000000000000306 ffff88023b2fe000
Call Trace:
 [<ffffffffa07e2850>] ? gfs2_check_blk_type+0x60/0x130 [gfs2]
 [<ffffffffa07e6a73>] gfs2_delete_inode+0x2f3/0x440 [gfs2]
 [<ffffffffa07e681d>] ? gfs2_delete_inode+0x9d/0x440 [gfs2]
 [<ffffffff810f0535>] ? call_rcu_sched+0x15/0x20
 [<ffffffffa07e6780>] ? gfs2_delete_inode+0x0/0x440 [gfs2]
 [<ffffffff811ac45e>] generic_delete_inode+0xde/0x1d0
 [<ffffffffa07c9010>] ? delete_work_func+0x0/0x90 [gfs2]
 [<ffffffff811ac5b5>] generic_drop_inode+0x65/0x80
 [<ffffffffa07e48c7>] gfs2_drop_inode+0x37/0x40 [gfs2]
 [<ffffffff811ab402>] iput+0x62/0x70
 [<ffffffffa07c9069>] delete_work_func+0x59/0x90 [gfs2]
 [<ffffffff81098100>] worker_thread+0x170/0x2a0
 [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f90>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e78e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 00 00 48 89 5d b0 e8 9d dd ff ff 85 c0 75 63 48 8b 45 b8 8b 4d c0 4c 89 e7 48 8b 18 8b 40 18 89 ca c1 ea 02 83 e1 03 89 d2 01 c9 <48> 03 43 28 0f b6 04 10 d3 f8 83 e0 03 44 39 f0 b8 8c ff ff ff 
RIP  [<ffffffffa07e28a3>] gfs2_check_blk_type+0xb3/0x130 [gfs2]
 RSP <ffff88043b047c60>
CR2: 0000000000000028
  • After a gfs2_grow a panic can be triggered some time afterwards:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
IP: [<ffffffffa061091c>] gfs2_glock_nq+0x18c/0x420 [gfs2]
PGD 63c1de067 PUD 63c1d3067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtual/block/dm-4/range
CPU 4 
Modules linked in: gfs2 dlm sctp libcrc32c configfs autofs4 dm_multipath uinput ipmi_devintf microcode power_meter acpi_ipmi ipmi_si ipmi_msghandler dcdbas joydev serio]

Pid: 4044, comm: bobs_accordion8 Not tainted 2.6.32-595.el6.all68n.x86_64 #1 Dell Inc. PowerEdge R815/06JC9T
RIP: 0010:[<ffffffffa061091c>]  [<ffffffffa061091c>] gfs2_glock_nq+0x18c/0x420 [gfs2]
RSP: 0018:ffff88063c617a28  EFLAGS: 00010292
RAX: 0000000000000000 RBX: ffff88023a1d3858 RCX: 00000000000014ef
RDX: 0000000000000003 RSI: 0000000000000046 RDI: ffffffffa0636a08
RBP: ffff88063c617a78 R08: 0000000000016d3f R09: 00000000fffffffb
R10: 0000000000000003 R11: 0000000000000001 R12: ffff88063b07b7d0
R13: ffff88063b07b7d0 R14: ffff88063b07b804 R15: ffff88023a1d3d58
FS:  00007f7d5e412700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000028 CR3: 000000063c1dd000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bobs_accordion8 (pid: 4044, threadinfo ffff88063c614000, task ffff88063b4e0040)
Stack:
 ffff88063c617a78 ffffffff8117c563 ffff88043ced9758 ffff88063af98000
<d> ffff88063b069728 ffff88063af98000 ffff88023a1d3840 ffff88023a1d3858
<d> 000000000000001d 000000000000000f ffff88063c617ab8 ffffffffa062c962
Call Trace:
 [<ffffffff8117c563>] ? kmem_cache_alloc_trace+0x1b3/0x1c0
 [<ffffffffa062c962>] gfs2_trans_begin+0xf2/0x190 [gfs2]
 [<ffffffffa0618469>] gfs2_write_begin+0x1d9/0x480 [gfs2]
 [<ffffffff8112bdb3>] generic_file_buffered_write+0x123/0x2e0
 [<ffffffff81081937>] ? current_fs_time+0x27/0x30
 [<ffffffff8112d810>] __generic_file_aio_write+0x260/0x490
 [<ffffffff8112dac8>] generic_file_aio_write+0x88/0x100
 [<ffffffffa061afa3>] gfs2_file_aio_write+0xf3/0x130 [gfs2]
 [<ffffffff8119804d>] ? __sb_end_write+0x3d/0x70
 [<ffffffff81195cca>] do_sync_write+0xfa/0x140
 [<ffffffff810a45f0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81243e9b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff81236b86>] ? security_file_permission+0x16/0x20
 [<ffffffff81195fc8>] vfs_write+0xb8/0x1a0
 [<ffffffff811974b6>] ? fget_light_pos+0x16/0x50
 [<ffffffff81196b01>] sys_write+0x51/0xb0

Environment

  • Red Hat Enterprise Linux Server 6 (with the High Availability Add On and Resilient Storage Add Ons)
  • A Global Filesystem 2(gfs2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.