A gfs2 filesystem had a withdraw: function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 428

Solution Verified - Updated -

Issue

A gfs2 filesystem experienced the below withdraw:

Apr  2 13:52:11 node42 kernel: GFS2: fsid=mycluster:shared_data.0: fatal: invalid metadata block
Apr  2 13:52:11 node42 kernel: GFS2: fsid=mycluster:shared_data.0:   bh = 16777216 (magic number)
Apr  2 13:52:11 node42 kernel: GFS2: fsid=mycluster:shared_data.0:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line = 428
Apr  2 13:52:11 node42 kernel: GFS2: fsid=mycluster:shared_data.0: about to withdraw this file system
Apr  2 13:52:11 node42 kernel: GFS2: fsid=mycluster:shared_data.0: telling LM to unmount
Apr  2 13:52:12 node42 kernel: GFS2: fsid=mycluster:shared_data.0: withdrawn
Apr  2 13:52:12 node42 kernel: CPU: 2 PID: 35139 Comm: oracle Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-1062.18.1.el7.x86_64 #1
Apr  2 13:52:12 node42 kernel: Hardware name: HP ProLiant DL380p Gen8, BIOS P70 05/24/2019
Apr  2 13:52:12 node42 kernel: Call Trace:
Apr  2 13:52:12 node42 kernel: [<ffffffff97f7b416>] dump_stack+0x19/0x1b
Apr  2 13:52:12 node42 kernel: [<ffffffffc0cbfe06>] gfs2_lm_withdraw+0x146/0x180 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0cc0125>] gfs2_meta_check_ii+0x45/0x50 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6a69>] gfs2_meta_indirect_buffer+0xe9/0x150 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c8fc34>] __fillup_metapath+0x44/0x90 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c8fcba>] fillup_metapath+0x3a/0x50 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c901fe>] gfs2_hole_size.isra.26+0x2ce/0x3a0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c907d0>] gfs2_iomap_get+0x500/0x520 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c932e8>] gfs2_block_map+0x1a8/0x290 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c90>] gfs2_get_block_direct+0x10/0x20 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a8e6a6>] do_blockdev_direct_IO+0xf96/0x20a0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a8f805>] __blockdev_direct_IO+0x55/0x60
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca717f>] gfs2_direct_IO+0x1df/0x220 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca7011>] ? gfs2_direct_IO+0x71/0x220 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff979bfad3>] generic_file_direct_write+0xd3/0x190
Apr  2 13:52:12 node42 kernel: [<ffffffff979bfdc7>] __generic_file_aio_write+0x237/0x400
Apr  2 13:52:12 node42 kernel: [<ffffffff979bffe9>] generic_file_aio_write+0x59/0xa0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca9bf1>] gfs2_file_aio_write+0xe1/0x310 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a4dac8>] ? __sb_start_write+0x58/0x120
Apr  2 13:52:12 node42 kernel: [<ffffffff97a4bd7e>] ? do_readv_writev+0x19e/0x260
Apr  2 13:52:12 node42 kernel: [<ffffffff97b04757>] ? security_file_permission+0x27/0xa0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca9b10>] ? gfs2_release+0x50/0x50 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97aa1ce3>] do_io_submit+0x3e3/0x8a0
Apr  2 13:52:12 node42 kernel: [<ffffffff97aa21b0>] SyS_io_submit+0x10/0x20
Apr  2 13:52:12 node42 kernel: [<ffffffff97f8dede>] system_call_fastpath+0x25/0x2a
Apr  2 13:52:12 node42 kernel: [<ffffffff97f8de21>] ? system_call_after_swapgs+0xae/0x146
Apr  2 13:52:12 node42 kernel: ------------[ cut here ]------------
Apr  2 13:52:12 node42 kernel: WARNING: CPU: 2 PID: 35139 at fs/buffer.c:1166 __brelse+0x2e/0x50
Apr  2 13:52:12 node42 kernel: VFS: brelse: Trying to free free buffer
Apr  2 13:52:12 node42 kernel: Modules linked in: gfs2 dlm mptctl mptbase ib_core scsi_transport_iscsi bridge stp llc ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw joydev gf128mul glue_helper ablk_helper cryptd pcspkr hpwdt lpc_ich hpilo ioatdma dca ses enclosure scsi_transport_sas ipmi_si ipmi_devintf wmi pcc_cpufreq ipmi_msghandler sg acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi dm_round_robin sd_mod crc_t10dif crct10dif_generic uas usb_storage mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt qla2xxx(OE) fb_sys_fops ttm ahci drm ata_piix libahci crct10dif_pclmul crct10dif_common crc32c_intel libata nvme_fc
Apr  2 13:52:12 node42 kernel: serio_raw dm_multipath nvme_fabrics nvme_core tg3(OE) scsi_transport_fc ptp drm_panel_orientation_quirks pps_core scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
Apr  2 13:52:12 node42 kernel: CPU: 2 PID: 35139 Comm: oracle Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-1062.18.1.el7.x86_64 #1
Apr  2 13:52:12 node42 kernel: Hardware name: HP ProLiant DL380p Gen8, BIOS P70 05/24/2019
Apr  2 13:52:12 node42 kernel: Call Trace:
Apr  2 13:52:12 node42 kernel: [<ffffffff97f7b416>] dump_stack+0x19/0x1b
Apr  2 13:52:12 node42 kernel: [<ffffffff9789bab8>] __warn+0xd8/0x100
Apr  2 13:52:12 node42 kernel: [<ffffffff9789bb3f>] warn_slowpath_fmt+0x5f/0x80
Apr  2 13:52:12 node42 kernel: [<ffffffff97a8269e>] __brelse+0x2e/0x50
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c8fcf6>] release_metapath+0x26/0x40 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c90183>] gfs2_hole_size.isra.26+0x253/0x3a0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c907d0>] gfs2_iomap_get+0x500/0x520 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0c932e8>] gfs2_block_map+0x1a8/0x290 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c90>] gfs2_get_block_direct+0x10/0x20 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a8e6a6>] do_blockdev_direct_IO+0xf96/0x20a0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a8f805>] __blockdev_direct_IO+0x55/0x60
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca717f>] gfs2_direct_IO+0x1df/0x220 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca6c80>] ? gfs2_meta_ra+0x1b0/0x1b0 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca7011>] ? gfs2_direct_IO+0x71/0x220 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff979bfad3>] generic_file_direct_write+0xd3/0x190
Apr  2 13:52:12 node42 kernel: [<ffffffff979bfdc7>] __generic_file_aio_write+0x237/0x400
Apr  2 13:52:12 node42 kernel: [<ffffffff979bffe9>] generic_file_aio_write+0x59/0xa0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca9bf1>] gfs2_file_aio_write+0xe1/0x310 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97a4dac8>] ? __sb_start_write+0x58/0x120
Apr  2 13:52:12 node42 kernel: [<ffffffff97a4bd7e>] ? do_readv_writev+0x19e/0x260
Apr  2 13:52:12 node42 kernel: [<ffffffff97b04757>] ? security_file_permission+0x27/0xa0
Apr  2 13:52:12 node42 kernel: [<ffffffffc0ca9b10>] ? gfs2_release+0x50/0x50 [gfs2]
Apr  2 13:52:12 node42 kernel: [<ffffffff97aa1ce3>] do_io_submit+0x3e3/0x8a0
Apr  2 13:52:12 node42 kernel: [<ffffffff97aa21b0>] SyS_io_submit+0x10/0x20
Apr  2 13:52:12 node42 kernel: [<ffffffff97f8dede>] system_call_fastpath+0x25/0x2a
Apr  2 13:52:12 node42 kernel: [<ffffffff97f8de21>] ? system_call_after_swapgs+0xae/0x146
Apr  2 13:52:12 node42 kernel: ---[ end trace 09d699ce5bfc0f6d ]---

Environment

  • Red Hat Enterprise Linux Server 7 or 8 (with the High Availability and Resilient Storage Add Ons)
  • A Global Filesystem 2 (gfs2)
  • Pacemaker

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content