A gfs2 withdrawal occurred: function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1169

Issue

• A gfs2 withdrawal occurred:

  Feb 15 04:18:56 node42 kernel: gfs2: fsid=cluster8:lv-data.4: fatal: filesystem consistency error
  Feb 15 04:18:56 node42 kernel: gfs2   inode = 61 1018716206
  Feb 15 04:18:56 node42 kernel: gfs2   function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1169
  Feb 15 04:18:56 node42 kernel: gfs2: fsid=cluster8:lv-data.4: about to withdraw this file system
  

• A gfs2 withdrawal occurred:

  Aug  4 19:14:33 node42 kernel: gfs2: fsid=rhel8cluster:gfs2-data.1: fatal: filesystem consistency error#012  inode = 2681 62542267#012  function = gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1181
  Aug  4 19:14:33 node42 kernel: gfs2: fsid=rhel8cluster:gfs2-data.1: about to withdraw this file system
  Aug  4 19:14:58 node42 kernel: watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kworker/12:0:1770816]
  Aug  4 19:14:58 node42 kernel: Modules linked in: tcp_diag udp_diag inet_diag mfe_fileaccess_100705139(OE) mfe_aac_100705139(OE) gfs2 dlm team_mode_activebackup nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject team nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vfat fat rpcrdma rdma_ucm ib_srpt ib_isert intel_rapl_msr intel_rapl_common iscsi_target_mod isst_if_common target_core_mod ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul qedr ipmi_ssif rapl ib_uverbs cdc_eem intel_cstate acpi_ipmi ib_core mei_me pcspkr usbnet ses intel_uncore ioatdma enclosure mii mei lpc_ich hpwdt hpilo ipmi_si dca ipmi_devintf wmi ipmi_msghandler acpi_tad acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c dm_snapshot dm_bufio dm_round_robin ext4 mbcache jbd2 sd_mod sg qla2xxx mgag200
  Aug  4 19:14:58 node42 kernel: drm_kms_helper qede nvme_fc syscopyarea nvme_fabrics sysfillrect qed nvme_core sysimgblt fb_sys_fops smartpqi t10_pi drm scsi_transport_fc scsi_transport_sas crc8 i2c_algo_bit dm_multipath dm_mirror dm_region_hash dm_log dm_mod crypto_user ansi_cprng cmac ccm xts des3_ede_x86_64 des_generic ghash_clmulni_intel crct10dif_pclmul crc32c_intel sha3_generic sha512_mb sha512_ssse3 sha512_generic sha256_mb sha1_mb mcryptd [last unloaded: tcrypt]
  Aug  4 19:14:58 node42 kernel: CPU: 12 PID: 1770816 Comm: kworker/12:0 Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-305.10.2.el8_4.x86_64 #1
  Aug  4 19:14:58 node42 kernel: Hardware name: HPE Synergy 480 Gen10/Synergy 480 Gen10 Compute Module, BIOS I42 05/24/2021
  Aug  4 19:14:58 node42 kernel: Workqueue: delete_workqueue delete_work_func [gfs2]
  Aug  4 19:14:58 node42 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x5b/0x1d0
  Aug  4 19:14:58 node42 kernel: Code: 6d f0 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 47 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 8b 37 81 fe 00 01 00
  Aug  4 19:14:58 node42 kernel: RSP: 0018:ffffb7554adffa98 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
  Aug  4 19:14:58 node42 kernel: RAX: 0000000000000101 RBX: ffff9ccbfafa4300 RCX: ffffffffc0ea4a20
  Aug  4 19:14:58 node42 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ccbfafa4300
  Aug  4 19:14:58 node42 kernel: RBP: ffff9cd38b409000 R08: 0000000000000001 R09: ffff9cc487803740
  Aug  4 19:14:58 node42 kernel: R10: 0000000000000000 R11: ffffffffa9458308 R12: ffffffffc0e48f90
  Aug  4 19:14:58 node42 kernel: R13: ffff9cd38b409000 R14: ffffffffc0e71200 R15: 0000000000000903
  Aug  4 19:14:58 node42 kernel: FS:  0000000000000000(0000) GS:ffff9cd39f680000(0000) knlGS:0000000000000000
  Aug  4 19:14:58 node42 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  Aug  4 19:14:58 node42 kernel: CR2: 00007f4d3c0b9fb8 CR3: 0000000d73410006 CR4: 00000000007706e0
  Aug  4 19:14:58 node42 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  Aug  4 19:14:58 node42 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  Aug  4 19:14:58 node42 kernel: PKRU: 55555554
  Aug  4 19:14:58 node42 kernel: Call Trace:
  Aug  4 19:14:58 node42 kernel: _raw_spin_lock+0x1c/0x20
  Aug  4 19:14:58 node42 kernel: lockref_get_not_dead+0x38/0x90
  Aug  4 19:14:58 node42 kernel: glock_hash_walk+0xa6/0xc0 [gfs2]
  Aug  4 19:14:58 node42 kernel: gfs2_flush_delete_work+0x14/0x20 [gfs2]
  Aug  4 19:14:58 node42 kernel: gfs2_make_fs_ro+0x2b/0x260 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? _cond_resched+0x15/0x30
  Aug  4 19:14:58 node42 kernel: ? gfs2_glock_wait+0x1b/0x80 [gfs2]
  Aug  4 19:14:58 node42 kernel: gfs2_withdraw.cold.13+0x140/0x4c1 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? gfs2_freeze_lock+0x29/0x60 [gfs2]
  Aug  4 19:14:58 node42 kernel: evict_unlinked_inode+0x283/0x2a0 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? out_of_line_wait_on_bit+0x91/0xb0
  Aug  4 19:14:58 node42 kernel: ? init_wait_var_entry+0x50/0x50
  Aug  4 19:14:58 node42 kernel: gfs2_evict_inode+0x3a1/0x6a0 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? gfs2_meta_read+0x260/0x310 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? __inode_wait_for_writeback+0x7f/0xf0
  Aug  4 19:14:58 node42 kernel: ? gfs2_evict_inode+0x1f7/0x6a0 [gfs2]
  Aug  4 19:14:58 node42 kernel: evict+0xd2/0x1a0
  Aug  4 19:14:58 node42 kernel: gfs2_inode_lookup+0x1e7/0x410 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? __switch_to_asm+0x41/0x70
  Aug  4 19:14:58 node42 kernel: ? __switch_to_asm+0x41/0x70
  Aug  4 19:14:58 node42 kernel: gfs2_lookup_by_inum+0x1c/0x50 [gfs2]
  Aug  4 19:14:58 node42 kernel: delete_work_func+0x93/0x1f0 [gfs2]
  Aug  4 19:14:58 node42 kernel: ? __switch_to+0x183/0x480
  Aug  4 19:14:58 node42 kernel: process_one_work+0x1a7/0x360
  Aug  4 19:14:58 node42 kernel: worker_thread+0x30/0x390
  Aug  4 19:14:58 node42 kernel: ? create_worker+0x1a0/0x1a0
  Aug  4 19:14:58 node42 kernel: kthread+0x116/0x130
  Aug  4 19:14:58 node42 kernel: ? kthread_flush_work_fn+0x10/0x10
  Aug  4 19:14:58 node42 kernel: ret_from_fork+0x35/0x40
  

• A gfs2 withdrawal occurred:

  Sep  5 01:30:49 node42 kernel: gfs2: fsid=cluster8:mydata.6: fatal: filesystem consistency error#012  inode = 17283 4652154#012  function =                 gfs2_dinode_dealloc, file = fs/gfs2/super.c, line = 1211
  Sep  5 01:30:49 node42 kernel: gfs2: fsid=cluster8:mydata.6: about to withdraw this file system
  Sep  5 01:33:12 node42 kernel: INFO: task kworker/16:3:3515634 blocked for more than 120 seconds.
  Sep  5 01:33:12 node42 kernel:      Tainted: P           OE    --------- -  - 4.18.0-305.3.1.el8_4.x86_64 #1
  Sep  5 01:33:12 node42 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep  5 01:33:12 node42 kernel: kworker/16:3    D    0 3515634      2 0x80004080
  Sep  5 01:33:12 node42 kernel: Workqueue: delete_workqueue delete_work_func [gfs2]
  Sep  5 01:33:12 node42 kernel: Call Trace:
  Sep  5 01:33:12 node42 kernel: __schedule+0x2c4/0x700
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x35/0x70
  Sep  5 01:33:12 node42 kernel: schedule+0x38/0xa0
  Sep  5 01:33:12 node42 kernel: schedule_timeout+0x246/0x2f0
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x41/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x35/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x41/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x35/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x41/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x35/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to_asm+0x41/0x70
  Sep  5 01:33:12 node42 kernel: ? __switch_to+0x183/0x480
  Sep  5 01:33:12 node42 kernel: wait_for_completion+0x97/0x100
  Sep  5 01:33:12 node42 kernel: flush_workqueue+0x14d/0x440
  Sep  5 01:33:12 node42 kernel: gfs2_make_fs_ro+0x19d/0x3f0 [gfs2]
  Sep  5 01:33:12 node42 kernel: ? gfs2_make_fs_ro+0x17d/0x3f0 [gfs2]
  Sep  5 01:33:12 node42 kernel: gfs2_withdraw.cold.11+0x11f/0x3ee [gfs2]
  Sep  5 01:33:12 node42 kernel: evict_unlinked_inode+0x283/0x2a0 [gfs2]
  Sep  5 01:33:12 node42 kernel: gfs2_evict_inode+0x3a1/0x6a0 [gfs2]
  Sep  5 01:33:12 node42 kernel: ? __wake_up_common_lock+0x89/0xc0
  Sep  5 01:33:12 node42 kernel: ? __inode_wait_for_writeback+0x7f/0xf0
  Sep  5 01:33:12 node42 kernel: ? gfs2_evict_inode+0x1f7/0x6a0 [gfs2]
  Sep  5 01:33:12 node42 kernel: evict+0xd2/0x1a0
  Sep  5 01:33:12 node42 kernel: gfs2_inode_lookup+0x1e7/0x410 [gfs2]
  Sep  5 01:33:12 node42 kernel: ? enqueue_entity+0xdc/0x640
  Sep  5 01:33:12 node42 kernel: gfs2_lookup_by_inum+0x1c/0x50 [gfs2]
  Sep  5 01:33:12 node42 kernel: delete_work_func+0x93/0x1f0 [gfs2]
  Sep  5 01:33:12 node42 kernel: process_one_work+0x1a7/0x360
  Sep  5 01:33:12 node42 kernel: worker_thread+0x30/0x390
  Sep  5 01:33:12 node42 kernel: ? create_worker+0x1a0/0x1a0
  Sep  5 01:33:12 node42 kernel: kthread+0x116/0x130
  Sep  5 01:33:12 node42 kernel: ? kthread_flush_work_fn+0x10/0x10
  Sep  5 01:33:12 node42 kernel: ret_from_fork+0x1f/0x40