Why virtual machines might fail to boot / observe data corruption after expanding a Gluster volume in RHHI / RHV environments?
Issue
- After expanding a Gluster volume in a RHHI setup or part of a RHV storage domain, the virtual machines or templates with images hosted in the expanded volume might start to observe data corruption.
-
The symptoms may vary: from VMs not starting to file system corruption observed in such VMs. Sample errors:
[ 34.152235] XFS (dm-2): Unmount and run xfs_repair [ 34.158182] XFS (dm-2): Metadata corruption detected at xfs_dir3_block_verify+0x3b/0xf0 [xfs], xfs_dir3_block block 0x5800340 [ 34.159032] XFS (dm-2): Unmount and run xfs_repair [ 34.164839] XFS (dm-2): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x5800340 len 8 error 117 [ 34.180656] XFS (dm-2): Metadata corruption detected at xfs_dir3_data_reada_verify+0x47/0x80 [xfs], xfs_dir3_data_reada block 0x54003f8 [ 34.181501] XFS (dm-2): Unmount and run xfs_repair [ 34.187750] XFS (dm-2): Metadata corruption detected at xfs_dir3_block_verify+0x3b/0xf0 [xfs], xfs_dir3_block block 0x54003f8 [ 34.188605] XFS (dm-2): Unmount and run xfs_repair OR [ +0.000688] XFS (dm-3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. [ +0.001570] XFS (dm-3): Metadata corruption detected at xfs_inode_buf_verify+0x14d/0x160 [xfs], xfs_inode block 0x3c048e0 xfs_inode_buf_verify [ +0.001193] XFS (dm-3): Unmount and run xfs_repair [ +0.000426] XFS (dm-3): First 128 bytes of corrupted metadata buffer:
-
What is causing this corruption? How to fix it?
Environment
- Red Hat Hyperconverged Infrastructure version 1.x
- Red Hat Virtualization version 4.x with Gluster setup as a storage domain
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.