A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7

Solution In Progress - Updated -

Issue

A kernel panic occurred after a gfs2 filesystem had a withdraw on RHEL 7.

[30165.991660] sd 7:0:3:3: Parameters changed
[30166.021772] sd 9:0:0:6: Parameters changed
[30166.022293] sd 7:0:3:6: Parameters changed
[30166.028221] sd 7:0:3:6: reservation conflict
[30166.029949] scsi_io_completion: 64 callbacks suppressed
[30166.031156] sd 9:0:0:6: reservation conflict
[30166.031161] blk_update_request: critical nexus error, dev sdbh, sector 94346824
[30166.031181] blk_update_request: critical nexus error, dev dm-62, sector 94346824
[30166.031626] sd 9:0:3:6: Parameters changed
[30166.037216] sd 9:0:3:6: reservation conflict
[30166.037221] blk_update_request: critical nexus error, dev sdej, sector 73900672
[30166.037235] blk_update_request: critical nexus error, dev dm-62, sector 73900672
[30166.037242] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: Error -52 writing to journal, jid=1
[30166.037250] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: error -52: withdrawing the file system to prevent further damage.
[30166.037251] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: about to withdraw this file system
[30166.037676] sd 7:0:2:6: Parameters changed
[30166.044159] sd 7:0:2:6: reservation conflict
[30166.044165] blk_update_request: critical nexus error, dev sdar, sector 73900696
[30166.044182] blk_update_request: critical nexus error, dev dm-62, sector 73900696
[30166.044188] GFS2: fsid=wspra01m0337:gfs2lock_apps.1: Error -52 writing to journal, jid=1
[30166.044505] sd 9:0:0:6: reservation conflict
[.....]
[30166.351079] sd 7:0:2:6: [sdar] tag#2 CDB: Write(10) 2a 00 00 a5 ea 00 00 04 00 00
[30166.351084] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: Error -52 writing to journal, jid=1
[30166.351092] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: error -52: withdrawing the file system to prevent further damage.
[30166.351094] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: about to withdraw this file system
[30166.351142] sd 7:0:2:6: reservation conflict
{.....]
[30166.582493] sd 9:0:0:6: reservation conflict
[30166.582499] sd 9:0:3:6: reservation conflict
[30166.582789] sd 7:0:2:6: reservation conflict
[30166.695838] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.882047] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.948853] GFS2: fsid=wspra01m0337:gfs2lock_apps_data.1: gfs2_evict_inode: -5
[30166.949259]  G:  s:EX n:2/335518 f:yIqb t:EX d:EX/0 a:1 v:0 r:-128 m:200
[30166.949616] ------------[ cut here ]------------
[30166.949917] kernel BUG at fs/gfs2/glock.c:250!
[30166.950250] invalid opcode: 0000 [#1] SMP
[30166.950591] Modules linked in: gfs2 dlm secvm2(POE) secfs2(POE) mptctl mptbase rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache team_mode_activebackup team dm_round_robin sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel dm_multipath lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr mgag200 i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_i801 hpilo lpc_ich drm_panel_orientation_quirks hpwdt ioatdma dca sg wmi ipmi_si ipmi_devintf ipmi_msghandler
[30166.954775]  pcc_cpufreq acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic qla2xxx bnx2x ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel tg3 serio_raw nvme_fc hpsa nvme_fabrics nvme_core mdio libcrc32c scsi_transport_fc ptp scsi_transport_sas pps_core scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[30166.957168] CPU: 23 PID: 7081 Comm: transform Kdump: loaded Tainted: P           OE  ------------   3.10.0-957.21.3.el7.x86_64 #1
[30166.958317] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/17/2018
[30166.958929] task: ffff9d977a2330c0 ti: ffff9d97a1728000 task.ti: ffff9d97a1728000
[30166.959566] RIP: 0010:[<ffffffffc104881b>]  [<ffffffffc104881b>] __gfs2_glock_put+0xdb/0xf0 [gfs2]
[30166.960239] RSP: 0018:ffff9d97a172bcc8  EFLAGS: 00010296
[30166.960917] RAX: 0000000000000000 RBX: ffff9db21ef497a8 RCX: 0000000000000006
[30166.961626] RDX: ffffffffc104a500 RSI: ffff9db21ef495f0 RDI: 0000000000000000
[30166.962562] RBP: ffff9d97a172bce8 R08: 0000000000000000 R09: ffff9db274e26f00
[30166.963282] R10: 000000000000292d R11: 0000000000000001 R12: ffff9db21ef495f0
[30166.964006] R13: ffff9dd1e2340000 R14: ffff9db21ef49620 R15: ffff9dd03ef45160
[30166.965019] FS:  00007f7902d70740(0000) GS:ffff9db27fdc0000(0000) knlGS:0000000000000000
[30166.965909] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[30166.966680] CR2: 00007f96d55c61d4 CR3: 0000002d1e37a000 CR4: 00000000001607e0
[30166.967481] Call Trace:
[30166.968292]  [<ffffffffc1048852>] gfs2_glock_put+0x22/0x30 [gfs2]
[30166.969121]  [<ffffffffc10488ff>] gfs2_holder_uninit+0x1f/0x40 [gfs2]
[30166.969947]  [<ffffffffc1049bda>] gfs2_glock_dq_uninit+0x1a/0x20 [gfs2]
[30166.970797]  [<ffffffffc105753e>] gfs2_create_inode+0x24e/0xe50 [gfs2]
[30166.971643]  [<ffffffffc1057b6c>] ? gfs2_create_inode+0x87c/0xe50 [gfs2]
[30166.972502]  [<ffffffffc1058175>] gfs2_mknod+0x35/0x40 [gfs2]
[30166.973372]  [<ffffffffae24ec57>] vfs_mknod+0xe7/0x160
[30166.974253]  [<ffffffffae2548c4>] SyS_mknodat+0x184/0x230
[30166.975130]  [<ffffffffae254970>] ? SyS_mknodat+0x230/0x230
[30166.976013]  [<ffffffffae25498d>] SyS_mknod+0x1d/0x20
[30166.976943]  [<ffffffffc0e32421>] m_sys_mknod+0x41/0x50 [secfs2]
[30166.977852]  [<ffffffffae775ddb>] system_call_fastpath+0x22/0x27
[30166.978774] Code: 48 8b 7b 08 48 83 c3 10 4c 89 e6 e8 f0 e7 33 ed 48 8b 03 48 85 c0 75 e8 eb b0 48 83 7b 60 00 74 a4 4c 89 e6 31 ff e8 65 f6 ff ff <0f> 0b 0f 1f 00 4c 89 e6 31 ff e8 56 f6 ff ff 0f 0b 0f 1f 40 00
[30166.980814] RIP  [<ffffffffc104881b>] __gfs2_glock_put+0xdb/0xf0 [gfs2]
[30166.981812]  RSP <ffff9d97a172bcc8>

Environment

  • Red Hat Enterprise Linux Server 7 (with the High Availability Add On and Resilient Storage Add Ons)
  • A Global Filesystem 2(gfs2)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In