RHEL7: XFS filesystem metadata corruption detected at xfs_bmbt_read_verify after reboot of cluster node "hang"

Issue

A system was found in a "hung" or unresponsive state. This is something that has been occurring frequently, at least once resulting seemingly from behavior in the cluster stack interaction with xfs.

After reboot, the system was up for about 3 hours, then the following xfs metadata corruption error was seen. The xfs_log_force errors repeated until the machine was rebooted. The filesystem that was corrupted was /var/log

[ 4227.296765] XFS (dm-6): Metadata corruption detected at xfs_bmbt_read_verify+0x79/0xc0 [xfs], block 0xafce8
[ 4227.296852] XFS (dm-6): Unmount and run xfs_repair
[ 4227.296890] XFS (dm-6): First 64 bytes of corrupted metadata buffer:
[ 4227.296939] ffff8807dd402000: 30 2e 30 30 20 20 20 20 20 30 2e 30 35 20 20 20  0.00     0.05   
[ 4227.297049] ffff8807dd402010: 20 30 2e 30 30 20 20 20 20 31 2e 31 35 20 20 20   0.00    1.15   
[ 4227.297112] ffff8807dd402020: 20 20 30 2e 30 30 20 20 20 20 32 31 2e 34 35 20    0.00    21.45 
[ 4227.297173] ffff8807dd402030: 20 20 20 33 37 2e 33 30 20 20 20 20 20 30 2e 30     37.30     0.0
[ 4227.297225] XFS (dm-6): metadata I/O error: block 0xafce8 ("xfs_trans_read_buf_map") error 117 numblks 8
[ 4227.297278] XFS (dm-6): xfs_do_force_shutdown(0x1) called from line 370 of file fs/xfs/xfs_trans_buf.c.  Return address = 0xffffffffa02579d6
[ 4227.298361] XFS (dm-6): I/O Error Detected. Shutting down filesystem
[ 4227.298396] XFS (dm-6): Please umount the filesystem and rectify the problem(s)
[ 4234.297491] XFS (dm-6): xfs_log_force: error -5 returned.
[ 4264.400771] XFS (dm-6): xfs_log_force: error -5 returned.
[ 4294.504139] XFS (dm-6): xfs_log_force: error -5 returned.
[ 4324.607493] XFS (dm-6): xfs_log_force: error -5 returned.