Why keepalived is not performing a failover upon network restart?
Environment
- Red Hat Enterprise Linux(RHEL) 7.
- Keepalived (All Versions).
- NetworkManager-1.0.4-9.el7 and above.
Issue
- When
NetworkManager
is running in the system and network restart is performed
Keepalived
is losingVIP
Keepalived
is not performing afailover
Resolution
- When
NetworkManager
is running in the system, it is NOT recommended to restartnetwork
service. - Use
ip link
to mark the interface link down and to verify thekeepalived
failover.
# ip link set down dev <interface-name>
Root Cause
-
Keepalived
tracks the interfacelink
status and performs afailover
if thelink
goesdown
. -
When there is
NetworkManager
running in the system andnetwork
service stops, it does not mark the interfacelink
status down. Sokeepalived
does not perform afailover
, aslink
status remains the same. -
However, when there is no NetworkManager and the network service stops, it marks the interface
link
statusdown
. As link goes down,keepalived
detects it and performs the failover.
Diagnostic Steps
- When there is no
NetworkManager
andnetwork
service stops, it brings down the interface, removes the IPs and marks the interfacelink
statusdown
.
Interface status when network service is running :-
# ethtool ens3 | grep -i "link det"
Link detected: yes
# ip a s ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:00:00:00:00:aa brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global dynamic ens3
valid_lft 3583sec preferred_lft 3583sec
inet 10.0.0.2/24 scope global secondary keepalived
valid_lft forever preferred_lft forever
# systemctl stop network
# ip a s ens3
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:00:00:00:00:aa brd ff:ff:ff:ff:ff:ff
# ethtool ens3 | grep -i "link det"
Link detected: no
- When there is
NetworkManager
and thenetwork
service stops, it brings down the interface, removes the IP butdoes not mark
the interfacelink
statusdown
.
Interface status when network service is running :-
# ethtool ens3 | grep -i "link det"
Link detected: yes
# ip a s ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:00:00:00:00:aa brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 brd 10.0.0.255 scope global dynamic ens3
valid_lft 3583sec preferred_lft 3583sec
inet 10.0.0.2/24 scope global secondary abc
valid_lft forever preferred_lft forever
# systemctl stop network
# ip a s ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:00:00:00:00:aa brd ff:ff:ff:ff:ff:ff
# ethtool ens3 | grep -i "link det"
Link detected: yes
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments