Why few nodes from CTDB cluster becomes UNHEALTHY when nodes participating in CTDB volumes were rebooted in Red Hat Gluster Storage

Issue

• 2 nodes out of 6 RHS nodes are not linked to CTDB

• Nodes become UNHEALTHY after reboot in CTDB

• CTDB status shows nodes UNHEALTHY after reboot.

• Why 2 nodes out of 6 node CTDB cluster becomes UNHEALTHY state when these 6 nodes are rebooted in Red Hat Gluster Storage 3.0 ?

~Snippet from terminal,

[root@rhs5 ~]# ctdb status
Number of nodes:6
pnn:0 x.x.x.x      OK
pnn:1 x.x.x.x       OK
pnn:2 x.x.x.x      OK
pnn:3 x.x.x.x      OK
pnn:4 x.x.x.x      UNHEALTHY (THIS NODE)
pnn:5 x.x.x.x      UNHEALTHY
Generation:2136328589
Size:6
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
hash:5 lmaster:5
Recovery mode:RECOVERY (1)
Recovery master:1

[root@rhs6 ~]# ctdb status
Number of nodes:6
pnn:0 x.x.x.x      OK
pnn:1 x.x.x.x      OK
pnn:2 x.x.x.x      OK
pnn:3 x.x.x.x      OK
pnn:4 x.x.x.x      UNHEALTHY
pnn:5 x.x.x.x      UNHEALTHY (THIS NODE)
Generation:1641607724
Size:6
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
hash:5 lmaster:5
Recovery mode:RECOVERY (1)
Recovery master:1

~Snippet from ctdb logs (/var/log/log.ctdb) as,

node rhs5:

2015/05/14 16:37:35.773567 [set_recmode:47635]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:37:39.077893 [set_recmode:47860]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:37:40.181628 [set_recmode:47923]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:37:41.173658 [set_recmode:47988]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!

node rhs6:

2015/05/14 16:36:52.238249 [set_recmode:32319]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:36:53.201144 [set_recmode:32387]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:36:54.518397 [set_recmode:32452]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!
2015/05/14 16:36:55.261483 [set_recmode:32517]: ERROR: recovery lock file /gluster/lock/lockfile not locked when recovering!