How to recover a file from split-brain on a Red Hat Gluster Storage volume ?

Solution Verified - Updated -

Issue

  • gluster volume heal VOLNAME info split-brain lists files that are affected with split-brain
  • After solving issues with XFS fs, gluster logs many errors related to split-brain
  • The client log for the volume shows messages indicating a split-brain:

    [afr-common.c:1215:afr_detect_self_heal_by_lookup_status] 0-my-replicate-0: entries are missing in lookup of /path/to/file.
    [afr-common.c:1341:afr_launch_self_heal] 0-my-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: /path/to/file, reason: lookup detected pending operations
    [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-my-replicate-0: path /path/to/file on subvolume my-client-0 => -1 (No such file or directory)
    [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-my-replicate-0: path /path/to/file on subvolume my-client-2 => -1 (No such file or directory)
    [afr-self-heal-data.c:769:afr_sh_data_fxattrop_fstat_done] 0-my-replicate-0: Unable to self-heal contents of '/path/to/file' (possible split-brain). Please delete the file from all but the preferred subvolume.
    [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-my-replicate-0: background  meta-data data entry missing-entry gfid self-heal failed on /path/to/file
    
  • The client log for the volume shows messages indicating conflicting entries:

    [afr-self-heal-common.c:1317:afr_sh_common_lookup_cbk] 0-my-replicate-1: Conflicting entries for /path/to/file
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-self-heal-common.c:1317:afr_sh_common_lookup_cbk] 0-my-replicate-1: Conflicting entries for /path/to/file
    [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-my-replicate-1: background entry self-heal failed on /path/to
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-common.c:1239:afr_detect_self_heal_by_iatt] 0-my-replicate-1: /path/to/file: gfid different on subvolume
    [afr-self-heal-common.c:1719:afr_sh_find_fresh_parents] 0-my-replicate-1: /path/to: Performing conservative merge
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-my-replicate-1: background meta-data data missing-entry self-heal failed on /path/to/file
    
  • Some entries may only reference a GFID.

[2013-10-29 00:43:11.375252] E [afr-self-heal-common.c:221:afr_sh_print_split_brain_log] 0-myvm-replicate-2: Unable to self-heal contents of '<gfid:d0ec7d7d-4f6c-4ac9-8304-4899294d7bcd>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 8170 ] [ 8172 0 ] ]

Environment

  • Red Hat Storage 2.0
  • Red Hat Storage 2.1
  • Red Hat Gluster Storage 3.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In