A kernel panic occurred on a RHEL 6 or RHEL 7 clustered node with gfs2 mount: "kernel BUG at fs/gfs2/rgrp.c:645!" or " kernel BUG at fs/gfs2/rgrp.c:649!"

Solution In Progress - Updated -

Issue

  • A kernel panic occurred on a RHEL 6 clustered node with gfs2 mounts:
------------[ cut here ]------------
kernel BUG at fs/gfs2/rgrp.c:645!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:06:00.1/host3/rport-3:0-9/target3:0:9/3:0:9:11/state
CPU 10 
Modules linked in: gfs2 dlm configfs mptctl mptbase nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc pcc_cpufreq arpt_mangle arptable_filter arp_tables ipv6 xfs exportfs microcode iTCO_wdt iTCO_vendor_support joydev serio_raw lpc_ich mfd_core hpilo hpwdt i7core_edac edac_core ipmi_devintf sg power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dm_round_robin sd_mod qla2xxx lpfc scsi_transport_fc scsi_tgt crc_t10dif be2net dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 27880, comm: nfsd Not tainted 2.6.32-642.1.1.el6.x86_64 #1 HP ProLiant BL460c G7
RIP: 0010:[<ffffffffa075bbe6>]  [<ffffffffa075bbe6>] gfs2_rsqa_delete+0x76/0x80 [gfs2]
RSP: 0018:ffff8817b31dfc60  EFLAGS: 00010202
RAX: 0000000000000020 RBX: ffff882d00784be0 RCX: 0000000000000000
RDX: ffff882fd5d34ea8 RSI: ffff882fd5d34f20 RDI: ffff882d00784f18
RBP: ffff8817b31dfc80 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000002000 R11: 0000000000000000 R12: ffff882d00784e0c
R13: ffff882d00784f68 R14: ffff8822b4866e00 R15: ffff8817d12e5980
FS:  0000000000000000(0000) GS:ffff8818608a0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007f317a150000 CR3: 0000000001a8d000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process nfsd (pid: 27880, threadinfo ffff8817b31dc000, task ffff8817e21d0040)
Stack:
 ffff882f09ef77c0 ffff882d00784be0 ffff882f09ef77c0 ffff882d00784be0
<d> ffff8817b31dfca0 ffffffffa074f30b ffff882f09ef77c0 0000000000000008
<d> ffff8817b31dfcf0 ffffffff8119b7d5 ffff882d00784be0 ffff8822b4866e00
Call Trace:
 [<ffffffffa074f30b>] gfs2_release+0x4b/0x60 [gfs2]
 [<ffffffff8119b7d5>] __fput+0xf5/0x210
 [<ffffffff8119b915>] fput+0x25/0x30
 [<ffffffffa067c343>] nfsd_write+0xf3/0x100 [nfsd]
 [<ffffffffa0684d5f>] nfsd3_proc_write+0xaf/0x140 [nfsd]
 [<ffffffffa0675405>] nfsd_dispatch+0xe5/0x230 [nfsd]
 [<ffffffffa055bd04>] svc_process_common+0x344/0x640 [sunrpc]
 [<ffffffff8106c720>] ? default_wake_function+0x0/0x20
 [<ffffffffa055c3a0>] svc_process+0x110/0x160 [sunrpc]
 [<ffffffffa0675c62>] nfsd+0xc2/0x160 [nfsd]
 [<ffffffffa0675ba0>] ? nfsd+0x0/0x160 [nfsd]
 [<ffffffff810a662e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code: 91 b4 ff ff 48 8b 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 0f 1f 00 48 8d bb f8 02 00 00 e8 44 ff ff ff 8b 83 68 03 00 00 85 c0 74 c6 <0f> 0b eb fe 66 0f 1f 44 00 00 55 48 89 e5 41 56 41 55 41 54 53 
RIP  [<ffffffffa075bbe6>] gfs2_rsqa_delete+0x76/0x80 [gfs2]
 RSP <ffff8817b31dfc60>
  • A kernel panic occurred on a RHEL 7 clustered node with gfs2 mounts:
[65473.060807] ------------[ cut here ]------------
[65473.065549] kernel BUG at fs/gfs2/rgrp.c:649!
[65473.070015] invalid opcode: 0000 [#1] SMP
[65473.074244] Modules linked in: gfs2 dlm mpt2sas raid_class mptctl mptbase xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 intel_powerclamp coretemp kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper iTCO_wdt ablk_helper cryptd iTCO_vendor_support dcdbas ipmi_devintf i7core_edac ses ipmi_ssif pcspkr enclosure dm_service_time lpc_ich scsi_transport_sas edac_core ipmi_si mfd_core sg shpchp ipmi_msghandler acpi_power_meter wmi acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc dm_multipath ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic i2c_algo_bit ixgbe drm_kms_helper mdio syscopyarea sysfillrect ptp sysimgblt fb_sys_fops crct10dif_pclmul qla2xxx pps_core ttm scsi_transport_fc ata_piix drm crct10dif_common crc32c_intel libata serio_raw i2c_core scsi_tgt dca megaraid_sas bnx2 fjes dm_mirror dm_region_hash dm_log dm_mod
[65473.176056] CPU: 22 PID: 2901 Comm: kworker/22:0 Not tainted 3.10.0-443.el7.x86_64 #1
[65473.184107] Hardware name: Dell Inc. PowerEdge R910/0KYD3D, BIOS 2.10.0 08/29/2013
[65473.191916] Workqueue: events delayed_fput
[65473.196128] task: ffff8818023a8ed0 ti: ffff88178231c000 task.ti: ffff88178231c000
[65473.203840] RIP: 0010:[<ffffffffa08b3389>]  [<ffffffffa08b3389>] gfs2_rsqa_delete+0x69/0x70 [gfs2]
[65473.213056] RSP: 0018:ffff88178231fd60  EFLAGS: 00010202
[65473.218477] RAX: 00000000000000c8 RBX: ffff88046ea97578 RCX: 0000000000000000
[65473.225721] RDX: ffff88066eb5a330 RSI: ffff88066eb5a3b0 RDI: ffff88066eb5a3ac
[65473.232969] RBP: ffff88178231fd78 R08: 0000000000000000 R09: 00000001802a0024
[65473.240213] R10: 000000000131d901 R11: ffffea006004c740 R12: ffff88046ea97440
[65473.247453] R13: ffff88046ea977c0 R14: ffff8817a00e2600 R15: ffff88178ccf1220
[65473.254699] FS:  0000000000000000(0000) GS:ffff88185f940000(0000) knlGS:0000000000000000
[65473.263016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[65473.268871] CR2: 00007fe454727044 CR3: 000000204e72b000 CR4: 00000000000007e0
[65473.276112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[65473.283357] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[65473.290599] Stack:
[65473.292716]  ffff8817c8cd9d00 ffff88046ea97440 ffff88046ea97440 ffff88178231fd98
[65473.300371]  ffffffffa08a50ff ffff8817c8cd9d00 0000000000000008 ffff88178231fde0
[65473.308034]  ffffffff811f5c39 ffff88046ea97440 ffff8817c8cd9d10 ffff8817c8cd9d00
[65473.315693] Call Trace:
[65473.318255]  [<ffffffffa08a50ff>] gfs2_release+0x3f/0x50 [gfs2]
[65473.324281]  [<ffffffff811f5c39>] __fput+0xe9/0x260
[65473.329269]  [<ffffffff811f5e5f>] delayed_fput+0xaf/0x130
[65473.334779]  [<ffffffff810a577b>] process_one_work+0x17b/0x470
[65473.340722]  [<ffffffff810a65b6>] worker_thread+0x126/0x410
[65473.346402]  [<ffffffff810a6490>] ? rescuer_thread+0x460/0x460
[65473.354963]  [<ffffffff810add6f>] kthread+0xcf/0xe0
[65473.359952]  [<ffffffff810adca0>] ? kthread_create_on_node+0x140/0x140
[65473.366590]  [<ffffffff8167ded8>] ret_from_fork+0x58/0x90
[65473.372102]  [<ffffffff810adca0>] ? kthread_create_on_node+0x140/0x140
[65473.378738] Code: 3e bf ff ff 5b 41 5c 41 5d 5d c3 0f 1f 80 00 00 00 00 49 8d bc 24 f0 02 00 00 e8 63 ff ff ff 41 8b 84 24 60 03 00 00 85 c0 74 c6 <0f> 0b 0f 1f 44 00 00 66 66 66 66 90 55 48 89 e5 41 56 4c 8d b7
[65473.398554] RIP  [<ffffffffa08b3389>] gfs2_rsqa_delete+0x69/0x70 [gfs2]

Environment

  • Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)
  • Red Hat Enterprise Linux Server 7 (with the High Availability and Resilient Storage Add Ons)
  • A Global Filesystem 2(GFS2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.