XFS Metadata corruption detected, xfs_repair not able to repair it

Solution Verified - Updated -

Issue

We are running 2 systems, they both have local disks where RHEL is installed, and they have access to a shared LUN (a blockdevice). On that LUN, we have a partition table, LVM physical volumes (PV), one volume group (not clustered (CLVM), but a normal volume group) and multiple volumes. These volumes use XFS. For one of these XFS filesystems, the following appeared:

hostname <kern.info> kernel:XFS (dm-5): Ending clean mount
hostname <kern.alert> kernel:XFS (dm-2): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], block 0x7d00001
hostname <kern.alert> kernel:XFS (dm-2): Unmount and run xfs_repair
hostname <kern.alert> kernel:XFS (dm-2): First 64 bytes of corrupted metadata buffer:
hostname <kern.alert> kernel:ffff881a4d5fa200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
hostname <kern.alert> kernel:ffff881a4d5fa210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
hostname <kern.alert> kernel:ffff881a4d5fa220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
hostname <kern.alert> kernel:ffff881a4d5fa230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
hostname <kern.alert> kernel:XFS (dm-2): metadata I/O error: block 0x7d00001 ("xfs_trans_read_buf_map") error 117 numblks 1 
hostname <kern.notice> kernel:XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 135 of file fs/xfs/xfs_bmap_util.c.  Return address = 0xffffffffa017a928
hostname <kern.alert> kernel:XFS (dm-2): Corruption of in-memory data detected.  Shutting down filesystem
hostname <kern.alert> kernel:XFS (dm-2): Please umount the filesystem and rectify the problem(s)
hostname <kern.warning> kernel:nr_pdflush_threads exported in /proc is scheduled for removal
hostname <kern.warning> kernel:XFS (dm-2): xfs_log_force: error -5 returned.

xfs_repair could not fix the corruption and we had to restore backup of the XFS filesystem from another storage volume. What is causing this? How can it be prevented?

Environment

  • Red Hat Enterprise Linux, versions 6, 7 and 8
  • LVM
  • shared storage between multiple systems

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content