A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7

Solution In Progress - Updated -

Issue

A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7.

[30165.991660] sd 7:0:3:3: Parameters changed
[30166.021772] sd 9:0:0:6: Parameters changed
[30166.022293] sd 7:0:3:6: Parameters changed
[30166.028221] sd 7:0:3:6: reservation conflict
[30166.029949] scsi_io_completion: 64 callbacks suppressed
[30166.031156] sd 9:0:0:6: reservation conflict
[30166.031161] blk_update_request: critical nexus error, dev sdbh, sector 94346824
[30166.031181] blk_update_request: critical nexus error, dev dm-62, sector 94346824
[30166.031626] sd 9:0:3:6: Parameters changed
[30166.037216] sd 9:0:3:6: reservation conflict
[30166.037221] blk_update_request: critical nexus error, dev sdej, sector 73900672
[30166.037235] blk_update_request: critical nexus error, dev dm-62, sector 73900672
[30166.037242] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: Error -52 writing to journal, jid=1
[30166.037250] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: error -52: withdrawing the file system to prevent further damage.
[30166.037251] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: about to withdraw this file system
[30166.037676] sd 7:0:2:6: Parameters changed
[30166.044159] sd 7:0:2:6: reservation conflict
[30166.044165] blk_update_request: critical nexus error, dev sdar, sector 73900696
[30166.044182] blk_update_request: critical nexus error, dev dm-62, sector 73900696
[30166.044188] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: Error -52 writing to journal, jid=1
[30166.044505] sd 9:0:0:6: reservation conflict
[.....]
[30166.351079] sd 7:0:2:6: [sdar] tag#2 CDB: Write(10) 2a 00 00 a5 ea 00 00 04 00 00
[30166.351084] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: Error -52 writing to journal, jid=1
[30166.351092] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: error -52: withdrawing the file system to prevent further damage.
[30166.351094] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: about to withdraw this file system
[30166.351142] sd 7:0:2:6: reservation conflict
{.....]
[30166.582493] sd 9:0:0:6: reservation conflict
[30166.582499] sd 9:0:3:6: reservation conflict
[30166.582789] sd 7:0:2:6: reservation conflict
[30166.695838] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.882047] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.948853] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.949259]  G:  s:EX n:2/335518 f:yIqb t:EX d:EX/0 a:1 v:0 r:-128 m:200
[30166.949616] ------------[ cut here ]------------
[30166.949917] kernel BUG at fs/gfs2/glock.c:250!
[30166.950250] invalid opcode: 0000 [#1] SMP
[30166.950591] Modules linked in: gfs2 dlm secvm2(POE) secfs2(POE) mptctl mptbase rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache team_mode_activebackup team dm_round_robin sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel dm_multipath lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr mgag200 i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_i801 hpilo lpc_ich drm_panel_orientation_quirks hpwdt ioatdma dca sg wmi ipmi_si ipmi_devintf ipmi_msghandler
[30166.954775]  pcc_cpufreq acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic qla2xxx bnx2x ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel tg3 serio_raw nvme_fc hpsa nvme_fabrics nvme_core mdio libcrc32c scsi_transport_fc ptp scsi_transport_sas pps_core scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[30166.957168] CPU: 23 PID: 7081 Comm: transform Kdump: loaded Tainted: P           OE  ------------   3.10.0-957.21.3.el7.x86_64 #1
[30166.958317] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
[30166.958929] task: ffff9d977a2330c0 ti: ffff9d97a1728000 task.ti: ffff9d97a1728000
[30166.959566] RIP: 0010:[<ffffffffc104881b>]  [<ffffffffc104881b>] __gfs2_glock_put+0xdb/0xf0 [gfs2]
[30166.960239] RSP: 0018:ffff9d97a172bcc8  EFLAGS: 00010296
[30166.960917] RAX: 0000000000000000 RBX: ffff9db21ef497a8 RCX: 0000000000000006
[30166.961626] RDX: ffffffffc104a500 RSI: ffff9db21ef495f0 RDI: 0000000000000000
[30166.962562] RBP: ffff9d97a172bce8 R08: 0000000000000000 R09: ffff9db274e26f00
[30166.963282] R10: 000000000000292d R11: 0000000000000001 R12: ffff9db21ef495f0
[30166.964006] R13: ffff9dd1e2340000 R14: ffff9db21ef49620 R15: ffff9dd03ef45160
[30166.965019] FS:  00007f7902d70740(0000) GS:ffff9db27fdc0000(0000) knlGS:0000000000000000
[30166.965909] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[30166.966680] CR2: 00007f96d55c61d4 CR3: 0000002d1e37a000 CR4: 00000000001607e0
[30166.967481] Call Trace:
[30166.968292]  [<ffffffffc1048852>] gfs2_glock_put+0x22/0x30 [gfs2]
[30166.969121]  [<ffffffffc10488ff>] gfs2_holder_uninit+0x1f/0x40 [gfs2]
[30166.969947]  [<ffffffffc1049bda>] gfs2_glock_dq_uninit+0x1a/0x20 [gfs2]
[30166.970797]  [<ffffffffc105753e>] gfs2_create_inode+0x24e/0xe50 [gfs2]
[30166.971643]  [<ffffffffc1057b6c>] ? gfs2_create_inode+0x87c/0xe50 [gfs2]
[30166.972502]  [<ffffffffc1058175>] gfs2_mknod+0x35/0x40 [gfs2]
[30166.973372]  [<ffffffffae24ec57>] vfs_mknod+0xe7/0x160
[30166.974253]  [<ffffffffae2548c4>] SyS_mknodat+0x184/0x230
[30166.975130]  [<ffffffffae254970>] ? SyS_mknodat+0x230/0x230
[30166.976013]  [<ffffffffae25498d>] SyS_mknod+0x1d/0x20
[30166.976943]  [<ffffffffc0e32421>] m_sys_mknod+0x41/0x50 [secfs2]
[30166.977852]  [<ffffffffae775ddb>] system_call_fastpath+0x22/0x27
[30166.978774] Code: 48 8b 7b 08 48 83 c3 10 4c 89 e6 e8 f0 e7 33 ed 48 8b 03 48 85 c0 75 e8 eb b0 48 83 7b 60 00 74 a4 4c 89 e6 31 ff e8 65 f6 ff ff <0f> 0b 0f 1f 00 4c 89 e6 31 ff e8 56 f6 ff ff 0f 0b 0f 1f 40 00
[30166.980814] RIP  [<ffffffffc104881b>] __gfs2_glock_put+0xdb/0xf0 [gfs2]
[30166.981812]  RSP <ffff9d97a172bcc8>

Environment

  • Red Hat Enterprise Linux Server 7 (with the High Availability Add On and Resilient Storage Add Ons)
  • A Global Filesystem 2(gfs2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content