NFS daemon is not running on any node

Solution In Progress - Updated -

Issue

NFS daemon is not running on any node due to locking issues by glusterd:

./rhgs-rdu2-c02-n05.storage.rdu2.redhat.com/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:[2017-06-16 09:16:48.467813] W [glusterd-locks.c:692:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_sm+0x29f) [0x7f723a239caf] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x657a2) [0x7f723a22b7a2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x37a) [0x7f723a2cbb8a] ) 0-management: Lock owner mismatch. Lock for vol ctdb held by xxxxxxxx-xxxx-xxxx-xxxxx
./rhgs-rdu2-c02-n05.storage.rdu2.redhat.com/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:[2017-06-16 09:16:48.473461] W [glusterd-locks.c:692:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_sm+0x29f) [0x7f723a239caf] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0x657a2) [0x7f723a22b7a2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock+0x37a) [0x7f723a2cbb8a] ) 0-management: Lock owner mismatch. Lock for vol gluster_shared_storage held by xxxxxxxx-xxxx-xxxx-xxxxx

Other scenarios:
Running volume rebalance/volume profile commands concurrently from all the nodes can cause one of the glusterd instance in a node to hold a volume lock for ever. Due to this, all the further commands on the same volume will fail with "another transaction is in progress" or "locking failed" error message. This is primarily seen when sosreport is executed on all the nodes at a same time.

Environment

glusterfs 3.*

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content