Gluster commands fail with lock error. Rebooting doesn't help.

Solution In Progress - Updated -

Issue

All gluster commands fail on a gluster node with errors like this:

[root@gluster01 ~]#  gluster volume statedump StP_GD128K_HSDR all
volume statedump: failed: Locking failed on cvhcbos020sds. Please check log file for details.

The glusterd.log log featured errors like these:

[2021-07-09 20:14:39.362390] E [MSGID: 106118] [glusterd-locks.c:411:glusterd_mgmt_v3_lock_entity] 0-management: Failed to acquire lock for vol volume01 on behalf of 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a.
[2021-07-09 20:14:39.362405] E [MSGID: 106145] [glusterd-locks.c:495:glusterd_multiple_mgmt_v3_lock] 0-management: Unable to lock all vol
[2021-07-09 20:14:39.362419] E [MSGID: 106118] [glusterd-mgmt-handler.c:77:glusterd_synctasked_mgmt_v3_lock] 0-management: Failed to acquire mgmt_v3 locks for 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a
[2021-07-09 20:14:39.362531] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking failed on gluster01. Please check log file for details.
[2021-07-09 20:14:39.362819] E [MSGID: 106118] [glusterd-mgmt.c:804:glusterd_mgmt_v3_initiate_lockdown] 0-management: Locking failed on gluster01. Please check log file for details.
[2021-07-09 20:14:39.362859] E [MSGID: 106119] [glusterd-mgmt.c:2439:glusterd_mgmt_v3_initiate_all_phases] 0-management: mgmt_v3 lockdown failed.
[2021-07-09 20:14:39.363593] W [glusterd-locks.c:807:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf4972) [0x7f522309d972] -->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf477e) [0x7f522309d77e] -->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf38d3) [0x7f522309c8d3] ) 0-management: Lock for vol volume01 not held
[2021-07-09 20:14:39.363609] E [MSGID: 106117] [glusterd-locks.c:342:glusterd_mgmt_v3_unlock_entity] 0-management: Failed to release lock for vol volume01 on behalf of 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a.

Rebooting a node usually clears locks, but not in this case. The errors continued after multiple reboots.

Environment

RHGS 3.5.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content