NetworkManager does not react to link state changes in RHEL7, 8 and 9 Server

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • NetworkManager

Issue

  • When a link goes down, NetworkManager does not remove its IP address. This means any routes associated with the interface are still present in the kernel routing table and connectivity over backup interfaces/gateways does not work as expected.
  • This is different than the behaviour of NetworkManager in RHEL 6.

Resolution

  • When ignore-carrier is set to * or yes, on link failure NetworkManager does not remove the IP address/route of the associated interface though the link state would be marked as Down.

  • If we want to modify the behavior, we can disable the NetworkManager ignore-carrier option:

    • Create a file in the /etc/NetworkManager/conf.d/ directory beginning with a number greater than 00, such as 99. with the following contents:
    # cat /etc/NetworkManager/conf.d/99-carrier.conf
[main]
ignore-carrier=no
    • Restart the NetworkManager service:
    # systemctl restart NetworkManager

Root Cause

  • By default, RHEL7, RHEL8 and RHEL9 Server includes the NetworkManager-config-server package. This package modifies the default NetworkManager behaviour to:

    1. Disable auto-configuration of all interfaces by enabling the no-auto-default option
    2. Disable monitoring of link state by enabling the ignore-carrier option
    # rpm -q --info NetworkManager-config-server
Name        : NetworkManager-config-server
Epoch       : 1
Version     : 1.8.0
Release     : 9.el7
Architecture: noarch
Install Date: Wed 02 Aug 2017 12:06:40 AM EDT
Group       : System Environment/Base
Size        : 1292
License     : GPLv2+
Signature   : RSA/SHA256, Wed 14 Jun 2017 05:28:29 AM EDT, Key ID 199e2f91fd431d51
Source RPM  : NetworkManager-1.8.0-9.el7.src.rpm
Build Date  : Wed 14 Jun 2017 04:26:49 AM EDT
Build Host  : ppc-045.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.gnome.org/projects/NetworkManager/
Summary     : NetworkManager config file for "server-like" defaults
Description :
This adds a NetworkManager configuration file to make it behave more
like the old "network" service. In particular, it stops NetworkManager
from automatically running DHCP on unconfigured ethernet devices, and
allows connections with static IP addresses to be brought up even on
ethernet devices with no carrier.

This package is intended to be installed by default for server
deployments.

  • The NetworkManager-config-server package provides a simple configuration file to override the default behaviour:

    # cat /usr/lib/NetworkManager/conf.d/00-server.conf
# This configuration file, when placed into into
# /etc/NetworkManager/conf.d changes NetworkManager's behavior to
# what's expected on "traditional UNIX server" type deployments.
#
# See "man NetworkManager.conf" for more information about these
# and other keys.
#
# Do not edit this file; it will be overwritten on upgrades. If you
# want to override the values here, or set additional values, you can
# do so by adding another file (eg, "99-local.conf") to this directory
# and setting keys there.

[main]
# Do not do automatic (DHCP/SLAAC) configuration on ethernet devices
# with no other matching connections.
no-auto-default=*

# Ignore the carrier (cable plugged in) state when attempting to
# activate static-IP connections.
ignore-carrier=*

  • These options are described in the NetworkManager.conf man page:

       no-auto-default
       Specify devices for which NetworkManager shouldn't
       create default wired connection (Auto eth0). By
       default, NetworkManager creates a temporary wired
       connection for any Ethernet device that is managed
       and doesn't have a connection configured. List a
       device in this option to inhibit creating the
       default connection for the device. May have the
       special value * to apply to all devices.

       When the default wired connection is deleted or
       saved to a new persistent connection by a plugin,
       the device is added to a list in the file
       /run/NetworkManager/no-auto-default.state to
       prevent creating the default connection for that
       device again.

       See the section called “Device List Format” for
       the syntax how to specify a device.

       Example:

           no-auto-default=00:22:68:5c:5d:c4,00:1e:65:ff:aa:ee
           no-auto-default=eth0,eth1
           no-auto-default=*

   ignore-carrier
       This setting is deprecated for the per-device
       setting ignore-carrier which overwrites this
       setting if specified (See ignore-carrier).
       Otherwise, it is a list of matches to specify for
       which device carrier should be ignored. See the
       section called “Device List Format” for the syntax
       how to specify a device. Note that master types
       like bond, bridge, and team ignore carrier by
       default. You can however revert that default using
       the "except:" specifier (or better, use the
       per-device setting instead of the deprecated
       setting).

Diagnostic Steps

  • Verify the current configuration of NetworkManager and see that ignore-carrier=* is set:

    [root@r74 ~]# NetworkManager --print-config
# NetworkManager configuration: /etc/NetworkManager/NetworkManager.conf (lib: 00-server.conf, 10-slaves-    order.conf)

[main]
# rc-manager=file
# auth-polkit=true
# dhcp=dhclient
no-auto-default=*
ignore-carrier=*
slaves-order=index
plugins=ifcfg-rh

[logging]
# backend=syslog
# audit=false

  • With ignore-carrier is set to * or yes, IP and route still exists with link state down:


     [root@localhost ~]# ip add s enp7s0
     3: enp7s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
         link/ether 52:54:00:b3:1e:e0 brd ff:ff:ff:ff:ff:ff
         inet 192.0.2.1/24 brd 192.0.2.255 scope global noprefixroute enp7s0
            valid_lft forever preferred_lft forever
         inet6 fe80::30b0:5a25:f708:b54d/64 scope link tentative noprefixroute 
            valid_lft forever preferred_lft forever

     [root@localhost ~]# ip route show | grep enp7s0
     default via 192.0.2.254 dev enp7s0 proto static metric 101 linkdown 
     192.0.2.0/24 dev enp7s0 proto kernel scope link src 192.0.2.1 metric 101 linkdown

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments