geo-rep status failing and snapshot creation failing on that particular volume, due to a lock held by one of the glusterd.

Issue

We cannot do a geo-rep status on that particular volume, due to a lock held by one of the glusterd.

We restarted that glusterd and was able to see the status and stopped the geo-rep session for that particular volume.

When we try to create the snapshot, we encountered pre-verification failure on node 4.

After restarting that glusterd, the pre-verification failure moved to node 2 and then finally node 1.

After toggling the glusterds, we managed to creat a snapshot without the complaint that goe-rep is not stopped.

Unfortunately on a second snapshot create, we encountered another failure to to "changelog notify" failure on bricks in node 1.

Restarting the glusterd resolved that issue, and we managed to create snapshots successfully after that.