A GFS2 filesystem threw an assertion which triggered a panic: "kernel BUG at fs/gfs2/inode.h:65!"

Solution Verified - Updated -

Issue

  • A GFS2 filesystem threw an assertion which triggered a panic: kernel BUG at fs/gfs2/inode.h:65! on RHEL 6:
GFS2: fsid=sasclus:lv_sasdata_prd.3: fatal assertion failed
------------[ cut here ]------------
kernel BUG at fs/gfs2/inode.h:65!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 1
Modules linked in: nfsd exportfs autofs4 gfs2 nfs lockd fscache auth_rpcgss nfs_acl sunrpc dlm configfs acpi_cpufreq freq_table mperf ipv6 iTCO_wdt iTCO_vendor_support sg microcode enic ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core joydev shpchp ext4 jbd2 mbcache dm_queue_length dm_round_robin sd_mod crc_t10dif fnic libfcoe libfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 8462, comm: delete_workqueu Not tainted 2.6.32-642.15.1.el6.x86_64 #1 Cisco Systems Inc UCSB-B22-M3/UCSB-B22-M3
RIP: 0010:[<ffffffffa045b2ae>]  [<ffffffffa045b2ae>] recursive_scan+0x73e/0x750 [gfs2]
RSP: 0018:ffff880c4d407a90  EFLAGS: 00010286
RAX: 0000000000000044 RBX: ffff88057b657800 RCX: 000000000000d967
RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246
RBP: ffff880c4d407b50 R08: 0000000000000000 R09: ffffffff81647b60
R10: 0000000000000001 R11: 0000000000000000 R12: ffff880c4fbb2638
R13: 000000001121f05f R14: 0000000000000008 R15: ffff880c4fbb3000
FS:  0000000000000000(0000) GS:ffff88002c220000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002ac3684b4ff0 CR3: 0000000001a8d000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process delete_workqueu (pid: 8462, threadinfo ffff880c4d404000, task ffff880c50d21520)
Stack:
 ffff880c4d407ab0 00000000ff86eedd ffff880c00000012 ffff880c000000b6
<d> 0000000100000000 ffff880c4fbb2018 ffff88057b657b88 0000000000000000
<d> ffff880c4fbb3000 ffff880100000000 ffff880c5ddbf000 ffff880c4d407c40
Call Trace:
 [<ffffffffa045b02b>] recursive_scan+0x4bb/0x750 [gfs2]
 [<ffffffff8112e6e7>] ? unlock_page+0x27/0x30
 [<ffffffff810a6727>] ? bit_waitqueue+0x17/0xd0
 [<ffffffffa045b3a5>] trunc_dealloc+0xe5/0x100 [gfs2]
 [<ffffffffa046246d>] ? gfs2_ea_dealloc+0x20d/0x5b0 [gfs2]
 [<ffffffffa045b3d0>] gfs2_file_dealloc+0x10/0x20 [gfs2]
 [<ffffffffa0485db8>] gfs2_delete_inode+0x1b8/0x440 [gfs2]
 [<ffffffffa0485c9d>] ? gfs2_delete_inode+0x9d/0x440 [gfs2]
 [<ffffffffa0485c00>] ? gfs2_delete_inode+0x0/0x440 [gfs2]
 [<ffffffff811b830e>] generic_delete_inode+0xde/0x1d0
 [<ffffffffa0468390>] ? delete_work_func+0x0/0x90 [gfs2]
 [<ffffffff811b8465>] generic_drop_inode+0x65/0x80
 [<ffffffffa0483cf7>] gfs2_drop_inode+0x37/0x40 [gfs2]
 [<ffffffff811b72b2>] iput+0x62/0x70
 [<ffffffffa046840f>] delete_work_func+0x7f/0x90 [gfs2]
 [<ffffffff8109fba0>] worker_thread+0x170/0x2a0
 [<ffffffff810a68a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8109fa30>] ? worker_thread+0x0/0x2a0
 [<ffffffff810a640e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6370>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 95 58 ff ff ff 43 8d 14 10 48 89 d0 48 f7 d8 48 89 45 88 e9 4b ff ff ff 48 8b 83 08 01 00 00 48 8b b8 90 02 00 00 e8 12 ca 02 00 <0f> 0b eb fe 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
RIP  [<ffffffffa045b2ae>] recursive_scan+0x73e/0x750 [gfs2]
 RSP <ffff880c4d407a90>
  • A GFS2 filesystem threw an assertion which triggered a panic: kernel BUG at fs/gfs2/inode.h:64! on RHEL 7:
[14959.087843] ------------[ cut here ]------------
[14959.088321] kernel BUG at fs/gfs2/inode.h:64!
[14959.088789] invalid opcode: 0000 [#1] SMP
[14959.089254] Modules linked in: sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag fileaccess_mod(OE) gfs2 dlm nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic
[14959.092108]  pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic ahci drm_kms_helper libahci syscopyarea ata_piix sysfillrect libata crct10dif_pclmul crct10dif_common sysimgblt fb_sys_fops ttm crc32c_intel drm vmxnet3 serio_raw i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
[14959.093389] CPU: 3 PID: 42903 Comm: kworker/3:1 Tainted: G           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
[14959.094048] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[14959.094721] Workqueue: delete_workqueue delete_work_func [gfs2]
[14959.095399] task: ffff8809cd34af70 ti: ffff8809d9e28000 task.ti: ffff8809d9e28000
[14959.096095] RIP: 0010:[<ffffffffc06ff749>]  [<ffffffffc06ff749>] gfs2_add_inode_blocks.part.17+0x14/0x16 [gfs2]
[14959.096795] RSP: 0018:ffff8809d9e2bb78  EFLAGS: 00010246
[14959.097488] RAX: 000000000000003e RBX: ffff880aa78b7440 RCX: 0000000000000000
[14959.098197] RDX: 0000000000000000 RSI: ffff880c0d6d38f8 RDI: ffff880c0d6d38f8
[14959.098898] RBP: ffff8809d9e2bb78 R08: 0000000000000000 R09: ffff880c0618bac0
[14959.099607] R10: 0000000000000837 R11: ffffc90006bcb9d8 R12: 0000000000000002
[14959.100402] R13: ffff88088a7c8790 R14: ffff88088a7c9000 R15: ffff880c052c7000
[14959.101143] FS:  0000000000000000(0000) GS:ffff880c0d6c0000(0000) knlGS:0000000000000000
[14959.101891] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[14959.102636] CR2: 00007fdab1763000 CR3: 00000009be7c8000 CR4: 00000000001607e0
[14959.103418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14959.104193] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14959.104939] Call Trace:
[14959.105681]  [<ffffffffc06d145b>] recursive_scan+0x79b/0x7a0 [gfs2]
[14959.106445]  [<ffffffffc06d157a>] trunc_dealloc+0x11a/0x140 [gfs2]
[14959.107206]  [<ffffffffc06dda4e>] ? gfs2_glock_wait+0x3e/0x80 [gfs2]
[14959.107959]  [<ffffffffc06df470>] ? gfs2_glock_nq+0x250/0x440 [gfs2]
[14959.108694]  [<ffffffffc06d31a0>] gfs2_file_dealloc+0x10/0x20 [gfs2]
[14959.109450]  [<ffffffffc06f94d8>] gfs2_evict_inode+0x288/0x470 [gfs2]
[14959.110206]  [<ffffffffc06f93a5>] ? gfs2_evict_inode+0x155/0x470 [gfs2]
[14959.110967]  [<ffffffff8122245c>] evict+0xac/0x180
[14959.111723]  [<ffffffff81222d6c>] iput+0xfc/0x190
[14959.112467]  [<ffffffffc06de33c>] delete_work_func+0x6c/0x80 [gfs2]
[14959.113220]  [<ffffffff810abe2f>] process_one_work+0x17f/0x440
[14959.113960]  [<ffffffff810acaf6>] worker_thread+0x126/0x3c0
[14959.114689]  [<ffffffff810ac9d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[14959.115479]  [<ffffffff810b4031>] kthread+0xd1/0xe0
[14959.116243]  [<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[14959.116994]  [<ffffffff816c0577>] ret_from_fork+0x77/0xb0
[14959.117729]  [<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[14959.118476] Code: c0 b9 ea ff ff ff e9 68 fd ff ff 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 8b 47 28 48 89 e5 48 8b b8 50 03 00 00 e8 d7 db ff ff <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 8b 47 28 48 89
[14959.120230] RIP  [<ffffffffc06ff749>] gfs2_add_inode_blocks.part.17+0x14/0x16 [gfs2]
[14959.121082]  RSP <ffff8809d9e2bb78>

Environment

  • Red Hat Enterprise Linux Server 6 (with the Resilient Storage Add-On)
  • Red Hat Enterprise Linux Server 7 (with the Resilient Storage Add-On)
  • A Global Filesystem 2 (gfs2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content