Why virtual machines might fail to boot / observe data corruption after expanding a Gluster volume in RHHI / RHV environments?

Solution Verified - Updated -

Issue

  • After expanding a Gluster volume in a RHHI setup or part of a RHV storage domain, the virtual machines or templates with images hosted in the expanded volume might start to observe data corruption.
  • The symptoms may vary: from VMs not starting to file system corruption observed in such VMs. Sample errors:

    [   34.152235] XFS (dm-2): Unmount and run xfs_repair
    [   34.158182] XFS (dm-2): Metadata corruption detected at xfs_dir3_block_verify+0x3b/0xf0 [xfs], xfs_dir3_block block 0x5800340
    [   34.159032] XFS (dm-2): Unmount and run xfs_repair
    [   34.164839] XFS (dm-2): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x5800340 len 8 error 117
    [   34.180656] XFS (dm-2): Metadata corruption detected at xfs_dir3_data_reada_verify+0x47/0x80 [xfs], xfs_dir3_data_reada block 0x54003f8
    [   34.181501] XFS (dm-2): Unmount and run xfs_repair
    [   34.187750] XFS (dm-2): Metadata corruption detected at xfs_dir3_block_verify+0x3b/0xf0 [xfs], xfs_dir3_block block 0x54003f8
    [   34.188605] XFS (dm-2): Unmount and run xfs_repair
    
    OR
    
    [  +0.000688] XFS (dm-3): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
    [  +0.001570] XFS (dm-3): Metadata corruption detected at xfs_inode_buf_verify+0x14d/0x160 [xfs], xfs_inode block 0x3c048e0 xfs_inode_buf_verify
    [  +0.001193] XFS (dm-3): Unmount and run xfs_repair
    [  +0.000426] XFS (dm-3): First 128 bytes of corrupted metadata buffer:
    
  • What is causing this corruption? How to fix it?

Environment

  • Red Hat Hyperconverged Infrastructure version 1.x
  • Red Hat Virtualization version 4.x with Gluster setup as a storage domain

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content