Is It Possible to Access the Data on a Replica-3 Gluster Volume, with 2 / 3 Nodes Down?

Issue

• There might be situations when, for some reason, 2 / 3 Gluster nodes are down and recovering them might not be fast. As client quorum is lost, the client mount points will return Transport endpoint is not connected( ENOTCONN ) and data will not be available. Is it possible to still access the volume information, taking advantage of the replica-3 configuration?

• One easy way could be setting the options cluster.quorum-count and cluster.quorum-type to reduce the minimum brick count to 1 for the required volumes. However, due to some recent changes in the AFR translator code, there's a possibility of a reverse healing occurring when the two bricks that are down, are brought up again. So, if there's no way to access the bricks in the nodes that are down to inspect the xattrs, setting the above quorum options might induce to reverse healing.

• In case of a 2 + 1 arbitrated volume, it's also possible to restore access to the data, but only in the case where one of the affected bricks by the outage is the arbiter brick. If both data bricks are down, there's no workaround possible.

• Please note that this is a best-effort solution, as Gluster is not designed to provide access to the data in case node quorum is lost. Before following the below steps, please contact Red Hat Support for further advice.