XFS filesystem metadata corruption detected at xfs_bmbt_read_verify after reboot of a RHEL7 hang cluster node

Issue

• System was found in a hung or unresponsive state. This is something that has been occurring frequently, at least once resulting seemingly from behavior in the cluster stack interaction with XFS.

• After reboot, the system was up for about 3 hours, then the following XFS metadata corruption error was seen. The xfs_log_force errors repeated until the machine was rebooted, and the corrupted filesystem was /var/log:

  XFS (dm-6): Metadata corruption detected at xfs_bmbt_read_verify+0x79/0xc0 [xfs], block 0xafce8
  XFS (dm-6): Unmount and run xfs_repair
  XFS (dm-6): First 64 bytes of corrupted metadata buffer:
  ffff8807dd402000: 30 2e 30 30 20 20 20 20 20 30 2e 30 35 20 20 20  0.00     0.05   
  ffff8807dd402010: 20 30 2e 30 30 20 20 20 20 31 2e 31 35 20 20 20   0.00    1.15   
  ffff8807dd402020: 20 20 30 2e 30 30 20 20 20 20 32 31 2e 34 35 20    0.00    21.45 
  ffff8807dd402030: 20 20 20 33 37 2e 33 30 20 20 20 20 20 30 2e 30     37.30     0.0
  XFS (dm-6): metadata I/O error: block 0xafce8 ("xfs_trans_read_buf_map") error 117 numblks 8
  XFS (dm-6): xfs_do_force_shutdown(0x1) called from line 370 of file fs/xfs/xfs_trans_buf.c.  Return address = 0xffffffffa02579d6
  XFS (dm-6): I/O Error Detected. Shutting down filesystem
  XFS (dm-6): Please umount the filesystem and rectify the problem(s)
  XFS (dm-6): xfs_log_force: error -5 returned.