Why is the comand 'gluster v heal VOLNAME split-brain <options>' failing with 'No such file or directory'(ENOENT)?
Issue
-
When reviewing the healing status of a volume, a file shows in split-brain:
root@s192.168.1.1:~ # gluster v heal test info Brick 192.168.1.1:/var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick file - Is in split-brain Status: Connected Number of entries: 1 Brick 192.168.1.2:/var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick file - Is in split-brain Status: Connected Number of entries: 1 Brick 192.168.1.3:/var/lib/heketi/mounts/vg_a7701d63845665dd54f506fb4709fb11/brick_6ae7f03af15eb6bf70c2c0087b2f3bc7/brick file - Is in split-brain Status: Connected Number of entries: 1 -
Checking the extended attributes of the file in each brick, they confirm that it's indeed in split-brain. Each brick is blaming the other two as having bad data:
root@s192.168.1.1:~ # getfattr -d -m. -e hex /var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick/file getfattr: Removing leading '/' from absolute path names # file: var/lib/heketi/mounts/vg_ecd4479c06e8df99c723a816bb0a2b0b/brick_a882ac489c316ea364c11ead26cb8af2/brick/file security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000010000000000000000 trusted.afr.test-client-1=0x000000040000000000000000 trusted.afr.test-client-2=0x000000040000000000000000 trusted.gfid=0x925b097494c64183ba799c7c3a8b7036 trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── root@s192.168.1.2:~ # getfattr -d -m. -e hex /var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick/file getfattr: Removing leading '/' from absolute path names # file: var/lib/heketi/mounts/vg_92da4ef89b7aaaed491403acb245f086/brick_15de9bb51b2975d4369a23aadc0090ac/brick/file security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000030000000000000000 trusted.afr.test-client-0=0x000000010000000000000000 trusted.afr.test-client-2=0x000000040000000000000000 trusted.gfid=0x925b097494c64183ba799c7c3a8b7036 trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── getfattr: Removing leading '/' from absolute path names # file: var/lib/heketi/mounts/vg_a7701d63845665dd54f506fb4709fb11/brick_6ae7f03af15eb6bf70c2c0087b2f3bc7/brick/file security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000030000000000000000 trusted.afr.test-client-0=0x000000010000000000000000 trusted.afr.test-client-1=0x000000040000000000000000 trusted.gfid=0x925b097494c64183ba799c7c3a8b7036 trusted.gfid2path.8c61ddf55ced1b46=0x32626334616664612d653535622d343832392d386538342d6263363364636334626435342f5f30303034302e77616c -
The steps for fixing files in split brain are covered in section 11.15.2 of the Gluster Administration Guide. However, when running the commands to fix this entry, they fail with the error
No such file or directory:root@s192.168.1.3:~ # gluster volume heal test split-brain bigger-file file Lookup failed on file:No such file or directory. Volume heal failed. -
Why is this so? How to correct this issue?
Environment
- Red Hat Gluster Storage version 3.5.2 and below
- Red Hat Openshift Container Storage version 3.11.5 and below
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.