Why is the comand 'gluster v heal VOLNAME split-brain <options>' failing with 'No such file or directory'(ENOENT)?

Solution Verified - Updated -

Issue

  • When reviewing the healing status of a volume, a file shows in split-brain:

    root@s192.168.1.1:~ # gluster v heal test info
    
    Brick 192.168.1.1:/var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick
    file - Is in split-brain
    Status: Connected
    Number of entries: 1
    
    Brick 192.168.1.2:/var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick
    file - Is in split-brain
    Status: Connected
    Number of entries: 1
    
    Brick 192.168.1.3:/var/lib/heketi/mounts/vg_a7701d63845665dd54f506fb4709fb11/brick_6ae7f03af15eb6bf70c2c0087b2f3bc7/brick
    file - Is in split-brain
    Status: Connected
    Number of entries: 1
    
  • Checking the extended attributes of the file in each brick, they confirm that it's indeed in split-brain. Each brick is blaming the other two as having bad data:

    root@s192.168.1.1:~ # getfattr -d -m. -e hex /var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick/file
    getfattr: Removing leading '/' from absolute path names
    # file: var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick/file
    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
    trusted.afr.dirty=0x000000010000000000000000
    trusted.afr.test-client-1=0x000000040000000000000000
    trusted.afr.test-client-2=0x000000040000000000000000
    trusted.gfid=0x925b097494c64183ba799c7c3a8b7036
    trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c
    ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    
    root@s192.168.1.2:~ # getfattr -d -m. -e hex /var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick/file
    getfattr: Removing leading '/' from absolute path names
    # file: var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick/file
    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
    trusted.afr.dirty=0x000000030000000000000000
    trusted.afr.test-client-0=0x000000010000000000000000
    trusted.afr.test-client-2=0x000000040000000000000000
    trusted.gfid=0x925b097494c64183ba799c7c3a8b7036
    trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c
    ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    
    getfattr: Removing leading '/' from absolute path names
    # file: var/lib/heketi/mounts/vg_a7701d63845665dd54f506fb4709fb11/brick_6ae7f03af15eb6bf70c2c0087b2f3bc7/brick/file
    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
    trusted.afr.dirty=0x000000030000000000000000
    trusted.afr.test-client-0=0x000000010000000000000000
    trusted.afr.test-client-1=0x000000040000000000000000
    trusted.gfid=0x925b097494c64183ba799c7c3a8b7036
    trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c
    
  • The steps for fixing files in split brain are covered in section 11.15.2 of the Gluster Administration Guide. However, when running the commands to fix this entry, they fail with the error No such file or directory:

    root@s192.168.1.3:~ # gluster volume heal test split-brain bigger-file file
    Lookup failed on file:No such file or directory.
    Volume heal failed.
    
  • Why is this so? How to correct this issue?

Environment

  • Red Hat Gluster Storage version 3.5.2 and below
  • Red Hat Openshift Container Storage version 3.11.5 and below

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content