Gluster commands fail with lock error. Rebooting doesn't help.
Issue
All gluster commands fail on a gluster node with errors like this:
[root@gluster01 ~]# gluster volume statedump StP_GD128K_HSDR all
volume statedump: failed: Locking failed on cvhcbos020sds. Please check log file for details.
The glusterd.log log featured errors like these:
[2021-07-09 20:14:39.362390] E [MSGID: 106118] [glusterd-locks.c:411:glusterd_mgmt_v3_lock_entity] 0-management: Failed to acquire lock for vol volume01 on behalf of 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a.
[2021-07-09 20:14:39.362405] E [MSGID: 106145] [glusterd-locks.c:495:glusterd_multiple_mgmt_v3_lock] 0-management: Unable to lock all vol
[2021-07-09 20:14:39.362419] E [MSGID: 106118] [glusterd-mgmt-handler.c:77:glusterd_synctasked_mgmt_v3_lock] 0-management: Failed to acquire mgmt_v3 locks for 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a
[2021-07-09 20:14:39.362531] E [MSGID: 106115] [glusterd-mgmt.c:117:gd_mgmt_v3_collate_errors] 0-management: Locking failed on gluster01. Please check log file for details.
[2021-07-09 20:14:39.362819] E [MSGID: 106118] [glusterd-mgmt.c:804:glusterd_mgmt_v3_initiate_lockdown] 0-management: Locking failed on gluster01. Please check log file for details.
[2021-07-09 20:14:39.362859] E [MSGID: 106119] [glusterd-mgmt.c:2439:glusterd_mgmt_v3_initiate_all_phases] 0-management: mgmt_v3 lockdown failed.
[2021-07-09 20:14:39.363593] W [glusterd-locks.c:807:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf4972) [0x7f522309d972] -->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf477e) [0x7f522309d77e] -->/usr/lib64/glusterfs/6.0/xlator/mgmt/glusterd.so(+0xf38d3) [0x7f522309c8d3] ) 0-management: Lock for vol volume01 not held
[2021-07-09 20:14:39.363609] E [MSGID: 106117] [glusterd-locks.c:342:glusterd_mgmt_v3_unlock_entity] 0-management: Failed to release lock for vol volume01 on behalf of 30c2c69d-9cc8-4b53-8bcc-7bf916b8851a.
Rebooting a node usually clears locks, but not in this case. The errors continued after multiple reboots.
Environment
RHGS 3.5.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.