How to recover a file from split-brain on a Red Hat Gluster Storage volume ?

Solution Verified - Updated -

Issue

  • gluster volume heal VOLNAME info split-brain lists files that are affected with split-brain
  • After solving issues with XFS fs, gluster logs many errors related to split-brain
  • The client log for the volume shows messages indicating a split-brain:

    [afr-common.c:1215:afr_detect_self_heal_by_lookup_status] 0-my-replicate-0: entries are missing in lookup of /path/to/file.
    [afr-common.c:1341:afr_launch_self_heal] 0-my-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: /path/to/file, reason: lookup detected pending operations
    [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-my-replicate-0: path /path/to/file on subvolume my-client-0 => -1 (No such file or directory)
    [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-my-replicate-0: path /path/to/file on subvolume my-client-2 => -1 (No such file or directory)
    [afr-self-heal-data.c:769:afr_sh_data_fxattrop_fstat_done] 0-my-replicate-0: Unable to self-heal contents of '/path/to/file' (possible split-brain). Please delete the file from all but the preferred subvolume.
    [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-my-replicate-0: background  meta-data data entry missing-entry gfid self-heal failed on /path/to/file
    
  • The client log for the volume shows messages indicating conflicting entries:

    [afr-self-heal-common.c:1317:afr_sh_common_lookup_cbk] 0-my-replicate-1: Conflicting entries for /path/to/file
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-self-heal-common.c:1317:afr_sh_common_lookup_cbk] 0-my-replicate-1: Conflicting entries for /path/to/file
    [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-my-replicate-1: background entry self-heal failed on /path/to
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-common.c:1239:afr_detect_self_heal_by_iatt] 0-my-replicate-1: /path/to/file: gfid different on subvolume
    [afr-self-heal-common.c:1719:afr_sh_find_fresh_parents] 0-my-replicate-1: /path/to: Performing conservative merge
    [afr-common.c:1462:afr_conflicting_iattrs] 0-my-replicate-1: /path/to/file: gfid differs on subvolume 1
    [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-my-replicate-1: background meta-data data missing-entry self-heal failed on /path/to/file
    
  • Some entries may only reference a GFID.

[2013-10-29 00:43:11.375252] E [afr-self-heal-common.c:221:afr_sh_print_split_brain_log] 0-myvm-replicate-2: Unable to self-heal contents of '<gfid:d0ec7d7d-4f6c-4ac9-8304-4899294d7bcd>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 8170 ] [ 8172 0 ] ]

Environment

  • Red Hat Storage 2.0
  • Red Hat Storage 2.1
  • Red Hat Gluster Storage 3.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content