XFS Metadata corruption detected, xfs_repair not able to repair it
Issue
We are running 2 systems, they both have local disks where RHEL is installed, and they have access to a shared LUN (a blockdevice). On that LUN, we have a partition table, LVM physical volumes (PV), one volume group (not clustered (CLVM), but a normal volume group) and multiple volumes. These volumes use XFS. For one of these XFS filesystems, the following appeared:
hostname <kern.info> kernel:XFS (dm-5): Ending clean mount
hostname <kern.alert> kernel:XFS (dm-2): Metadata corruption detected at xfs_agf_read_verify+0x70/0x120 [xfs], block 0x7d00001
hostname <kern.alert> kernel:XFS (dm-2): Unmount and run xfs_repair
hostname <kern.alert> kernel:XFS (dm-2): First 64 bytes of corrupted metadata buffer:
hostname <kern.alert> kernel:ffff881a4d5fa200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
hostname <kern.alert> kernel:ffff881a4d5fa210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
hostname <kern.alert> kernel:ffff881a4d5fa220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
hostname <kern.alert> kernel:ffff881a4d5fa230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
hostname <kern.alert> kernel:XFS (dm-2): metadata I/O error: block 0x7d00001 ("xfs_trans_read_buf_map") error 117 numblks 1
hostname <kern.notice> kernel:XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 135 of file fs/xfs/xfs_bmap_util.c. Return address = 0xffffffffa017a928
hostname <kern.alert> kernel:XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem
hostname <kern.alert> kernel:XFS (dm-2): Please umount the filesystem and rectify the problem(s)
hostname <kern.warning> kernel:nr_pdflush_threads exported in /proc is scheduled for removal
hostname <kern.warning> kernel:XFS (dm-2): xfs_log_force: error -5 returned.
xfs_repair could not fix the corruption and we had to restore backup of the XFS filesystem from another storage volume. What is causing this? How can it be prevented?
Environment
- Red Hat Enterprise Linux, versions 6, 7 and 8
- LVM
- shared storage between multiple systems
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.