Keepalived is losing VIP upon "nmcli con up <con-name>" and failover does not occur
Environment
- Red Hat Enterprise Linux 7
- Keepalived-1.x.x
Issue
After modifying a NetworkManager profile via nmcli, NetworkManager deletes the VIP address and does not reassign the VIP to the configured interface. As as consequence, communication to the VIP address is lost.
Resolution
If keepalived is in use, the system administrator should disable NetworkManager and revert back to initscripts to maintain interface configurations.
However, a workaround does exist that leverages NetworkManagers dispatcher scripts. Which can be used to inject events inline with device events. As NetworkManager ships with environment variables can be used in a script (A list of these variables can be found in man 8 NetworkManager in section DISPATCHER SCRIPTS).
For example, put-down.sh is created in /etc/NetworkManager/pre-down.d/put-down.sh, then symlinked to /etc/NetworkManager/pre-down.d/put-down.sh.If a connection profile is activated put-down.sh will be activated inline.
Disclaimer: The put-down.sh shell script is provided as is by Red Hat support to show proof of concept,not to be implemented in production.
% put-down.sh
1 #!/bin/bash
2 # author: openpgp:0x77604A02E1292C5A
3
4 set -xeuo pipefail
5 shopt -s nullglob
6
7 advrt_int=1
8 iface=${DEVICE_IP_IFACE:-}
9 act=${NM_DISPATCHER_ACTION:-}
10
11 function _go_set(){
12 local ifa=${1:-}
13 local op=${2:-}
14
15 logger info "[$$] $0: DEVICE_IP_IFACE: ${ifa} NM_DISPATCHER_ACTION: ${op}"
16
17 case ${op} in
18 down)
19 ip link set ${ifa} down
20 ;;
21 up)
22 ip link set ${ifa} down
23 sleep ${advrt}
24 ip link set ${ifa} up
25 ;;
26 esac
27 return 0
28 }
29
30 function main(){
31 declare -a argc=( "$#" )
32 declare -a argv=( "$@" )
33 local ifac=${argv[0]:-}
34 local ac=${argv[1]:-}
35
36 _go_set ${ifac} ${ac}
37
38 return 0
39 }
40
41 main ${iface} ${act}
Root Cause
keepalived and NetworkManager do not commingle their states nor do they communicate in any inter-process communication.NetworkManager will only configure an interface with the defined values in its connection or device profiles.keepalived will only monitor the state of the link but not the the state of vip address.
Commit 979727e5db1f0307149b2932267ed214ecd0850d added the ability to track the the vip address but is not available until keepalived-2.0.0. Releases prior to 1.x.x do not have this capability and will not be back ported.
Commit 979727e5db1f0307149b2932267ed214ecd0850d
}
}
/* Add each VIP/eVIP's interface to the interface list */
LIST_FOREACH(vrrp->vip, vip, e) {
if (!vip->ifp)
vip->ifp = vrrp->ifp;
add_vrrp_to_interface(vrrp, vip->ifp, VRRP_NOT_TRACK_IF, false);
}
LIST_FOREACH(vrrp->evip, vip, e) {
if (!vip->ifp)
vip->ifp = vrrp->ifp;
add_vrrp_to_interface(vrrp, vip->ifp, VRRP_NOT_TRACK_IF, false);
}
/* In case of VRRP SYNC, we have to carefully check that we are
* not running floating priorities on any VRRP instance, unless
* sgroup_tracking_weight is set.
Additional Articles for Reference
1. Why keepalived is not performing a failover upon network restart
2. NetworkManager service is not compatible with Load Balancer
3. Bug 1842706 - keepalived vrrp address lost after nmcli modication - CLOSED WONTFIX
Diagnostic Steps
- Modify the
NetworkManagerprofile where the VIP address is assigned to vianmcli con mod .... Then reactivate the connection profile on the Master node. Once completed you will notice the VIP address has been removed and a fail-over did not occur.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments