Keepalived is losing VIP upon "nmcli con up <con-name>" and failover does not occur

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Keepalived-1.x.x

Issue

After modifying a NetworkManager profile via nmcli, NetworkManager deletes the VIP address and does not reassign the VIP to the configured interface. As as consequence, communication to the VIP address is lost.

Resolution

If keepalived is in use, the system administrator should disable NetworkManager and revert back to initscripts to maintain interface configurations.

However, a workaround does exist that leverages NetworkManagers dispatcher scripts. Which can be used to inject events inline with device events. As NetworkManager ships with environment variables can be used in a script (A list of these variables can be found in man 8 NetworkManager in section DISPATCHER SCRIPTS).

For example, put-down.sh is created in /etc/NetworkManager/pre-down.d/put-down.sh, then symlinked to /etc/NetworkManager/pre-down.d/put-down.sh.If a connection profile is activated put-down.sh will be activated inline.

Disclaimer: The put-down.sh shell script is provided as is by Red Hat support to show proof of concept,not to be implemented in production.

% put-down.sh
  1 #!/bin/bash
  2 # author: openpgp:0x77604A02E1292C5A
  3 
  4 set -xeuo pipefail
  5 shopt -s nullglob
  6 
  7 advrt_int=1
  8 iface=${DEVICE_IP_IFACE:-}
  9 act=${NM_DISPATCHER_ACTION:-}
 10 
 11 function _go_set(){
 12  local ifa=${1:-}
 13  local op=${2:-}
 14 
 15  logger info "[$$] $0: DEVICE_IP_IFACE: ${ifa} NM_DISPATCHER_ACTION: ${op}"
 16 
 17         case ${op} in
 18             down)
 19                 ip link set ${ifa} down
 20                 ;;
 21               up)
 22                 ip link set ${ifa} down
 23                 sleep ${advrt}
 24                 ip link set ${ifa} up
 25                ;;
 26         esac
 27    return 0
 28 }
 29 
 30 function main(){
 31  declare -a argc=( "$#" )
 32  declare -a argv=( "$@" )
 33  local ifac=${argv[0]:-}
 34  local ac=${argv[1]:-}
 35 
 36   _go_set ${ifac} ${ac}
 37 
 38  return 0
 39 }
 40 
 41 main ${iface} ${act}

Root Cause

keepalived and NetworkManager do not commingle their states nor do they communicate in any inter-process communication.NetworkManager will only configure an interface with the defined values in its connection or device profiles.keepalived will only monitor the state of the link but not the the state of vip address.

Commit 979727e5db1f0307149b2932267ed214ecd0850d added the ability to track the the vip address but is not available until keepalived-2.0.0. Releases prior to 1.x.x do not have this capability and will not be back ported.

Commit 979727e5db1f0307149b2932267ed214ecd0850d

                                   }
    }

    /* Add each VIP/eVIP's interface to the interface list */
    LIST_FOREACH(vrrp->vip, vip, e) {
        if (!vip->ifp)
            vip->ifp = vrrp->ifp;
        add_vrrp_to_interface(vrrp, vip->ifp, VRRP_NOT_TRACK_IF, false);
    }
    LIST_FOREACH(vrrp->evip, vip, e) {
        if (!vip->ifp)
            vip->ifp = vrrp->ifp;
        add_vrrp_to_interface(vrrp, vip->ifp, VRRP_NOT_TRACK_IF, false);
    }

    /* In case of VRRP SYNC, we have to carefully check that we are
     * not running floating priorities on any VRRP instance, unless
     * sgroup_tracking_weight is set.

Additional Articles for Reference
1. Why keepalived is not performing a failover upon network restart
2. NetworkManager service is not compatible with Load Balancer
3. Bug 1842706 - keepalived vrrp address lost after nmcli modication - CLOSED WONTFIX

Diagnostic Steps

  • Modify the NetworkManager profile where the VIP address is assigned to via nmcli con mod .... Then reactivate the connection profile on the Master node. Once completed you will notice the VIP address has been removed and a fail-over did not occur.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments