Why CTDB not able to release a public IP address from a VLAN-Interface when it fails in Red Hat Gluster Storage 3.1 ?
Environment
- Red Hat Gluster Storage 3.1
- Red Hat Enterprise Linux Server 7.2
- ctdb2.5-2.5.5-7.el7rhgs.x86_64
Issue
- Why CTDB not able to release a 'public' IP address from a VLAN-Interface when it fails ?
~Snippet from logs,
2016/02/01 16:09:13.989176 [recoverd:23371]: Trigger IP reallocation
2016/02/01 16:09:13.989267 [23178]: server/ctdb_recover.c:999 startrecovery eventscript has been invoked
2016/02/01 16:09:14.085613 [recoverd:23371]: Takeover run starting
2016/02/01 16:09:14.085973 [23178]: Release of IP "IP-address-1"/24 on interface bond0.601 node:1
2016/02/01 16:09:14.086155 [23178]: Release of IP "IP-address-2"/24 on interface bond0.601 node:1
2016/02/01 16:09:14.107910 [23178]: 10.interface: WARNING: Public IP "IP-address-1" hosted on interface bond0.601@bond0 but VNN says bond0.601
2016/02/01 16:09:14.108956 [23178]: 10.interface: WARNING: Public IP "IP-address-2" hosted on interface bond0.601@bond0 but VNN says bond0.601
2016/02/01 16:09:14.116334 [23178]: 10.interface: /etc/ctdb/functions: line 193: /proc/sys/net/ipv4/conf/bond0.601@bond0/promote_secondaries: No such file or directory <<============
2016/02/01 16:09:14.118378 [23178]: 10.interface: Cannot find device "bond0.601@bond0" <<===============
2016/02/01 16:09:14.118495 [23178]: 10.interface: Failed to del "IP-address-1" on dev bond0.601@bond0
2016/02/01 16:09:14.118818 [23178]: 10.interface: /etc/ctdb/functions: line 193: /proc/sys/net/ipv4/conf/bond0.601@bond0/promote_secondaries: No such file or directory
2016/02/01 16:09:14.120404 [23178]: IP "IP-address-1" still hosted during release IP callback, failing
Resolution
-
This issue is fixed in
ctdb-4.2.4version -
Update the ctdb package after enabling the samba repo by following command as,
# subscription-manager repos --enable=rh-gluster-3-samba-for-rhel-7-server-rpms
Root Cause
This is actually a bug in the functionip_maskbits_iface(/etc/ctdb/functions). A fix is applied in the samba git repository. The fix just strips the '@' sign and everything behind it from the functions output.
Diagnostic Steps
Reproducing steps:
1. Start first CTDB node
2. Four 'public' addresses are all assigned to the first CTDB node
3. start CTDB on another server (joins the CTDB cluster)
4. Two of the 4 public address will move from the first node to the new joining node
5. failure: first node is unable to remove the IP addresses from it's VLAN interface
One will view the below ctdb package version from gluster channel as,
# yum list installed |grep ctdb
ctdb.x86_64 4.2.4-9.1.el7rhgs @rh-gluster-3-samba-for-rhel-7-server-rpms
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
