NFS daemon is not running on any node
Issue
NFS daemon is not running on any node due to locking issues by glusterd:
./rhgs-rdu2-c02-n05.storage.rdu2.redhat.com/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:[2017-06-16 09:16:48.467813] W [glusterd-locks.c:692:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_sm+0x29f) [0x7f723a239caf] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x657a2) [0x7f723a22b7a2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x37a) [0x7f723a2cbb8a] ) 0-management: Lock owner mismatch. Lock for vol ctdb held by xxxxxxxx-xxxx-xxxx-xxxxx
./rhgs-rdu2-c02-n05.storage.rdu2.redhat.com/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:[2017-06-16 09:16:48.473461] W [glusterd-locks.c:692:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_sm+0x29f) [0x7f723a239caf] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x657a2) [0x7f723a22b7a2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x37a) [0x7f723a2cbb8a] ) 0-management: Lock owner mismatch. Lock for vol gluster_shared_storage held by xxxxxxxx-xxxx-xxxx-xxxxx
Other scenarios:
Running volume rebalance/volume profile commands concurrently from all the nodes can cause one of the glusterd instance in a node to hold a volume lock for ever. Due to this, all the further commands on the same volume will fail with "another transaction is in progress" or "locking failed" error message. This is primarily seen when sosreport is executed on all the nodes at a same time.
Environment
glusterfs 3.*
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.