NetworkManager does not react to link state changes in RHEL 7 Server

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • NetworkManager

Issue

  • When a link goes down, NetworkManager does not remove its IP address. This means any routes associated with the interface are still present in the kernel routing table and connectivity over backup interfaces/gateways does not work as expected.
  • This is different than the behaviour of NetworkManager in RHEL 6.

Resolution

  • Disable the NetworkManager ignore-carrier option:

    • Create a file in the /etc/NetworkManager/conf.d/ directory beginning with a number greater than 00, such as 99. with the following contents:
    # cat /etc/NetworkManager/conf.d/99-carrier.conf
    [main]
    ignore-carrier=no
    
    • Restart the NetworkManager service:
    # systemctl restart NetworkManager
    

Root Cause

  • By default, RHEL 7 Server includes the NetworkManager-config-server package. This package modifies the default NetworkManager behaviour to:

    1. Disable auto-configuration of all interfaces by enabling the no-auto-default option
    2. Disable monitoring of link state by enabling the ignore-carrier option
    # rpm -q --info NetworkManager-config-server
    Name        : NetworkManager-config-server
    Epoch       : 1
    Version     : 1.8.0
    Release     : 9.el7
    Architecture: noarch
    Install Date: Wed 02 Aug 2017 12:06:40 AM EDT
    Group       : System Environment/Base
    Size        : 1292
    License     : GPLv2+
    Signature   : RSA/SHA256, Wed 14 Jun 2017 05:28:29 AM EDT, Key ID 199e2f91fd431d51
    Source RPM  : NetworkManager-1.8.0-9.el7.src.rpm
    Build Date  : Wed 14 Jun 2017 04:26:49 AM EDT
    Build Host  : ppc-045.build.eng.bos.redhat.com
    Relocations : (not relocatable)
    Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
    Vendor      : Red Hat, Inc.
    URL         : http://www.gnome.org/projects/NetworkManager/
    Summary     : NetworkManager config file for "server-like" defaults
    Description :
    This adds a NetworkManager configuration file to make it behave more
    like the old "network" service. In particular, it stops NetworkManager
    from automatically running DHCP on unconfigured ethernet devices, and
    allows connections with static IP addresses to be brought up even on
    ethernet devices with no carrier.
    
    This package is intended to be installed by default for server
    deployments.
    
  • The NetworkManager-config-server package provides a simple configuration file to override the default behaviour:

    # cat /usr/lib/NetworkManager/conf.d/00-server.conf
    # This configuration file, when placed into into
    # /etc/NetworkManager/conf.d changes NetworkManager's behavior to
    # what's expected on "traditional UNIX server" type deployments.
    #
    # See "man NetworkManager.conf" for more information about these
    # and other keys.
    #
    # Do not edit this file; it will be overwritten on upgrades. If you
    # want to override the values here, or set additional values, you can
    # do so by adding another file (eg, "99-local.conf") to this directory
    # and setting keys there.
    
    [main]
    # Do not do automatic (DHCP/SLAAC) configuration on ethernet devices
    # with no other matching connections.
    no-auto-default=*
    
    # Ignore the carrier (cable plugged in) state when attempting to
    # activate static-IP connections.
    
  • These options are described in the NetworkManager.conf man page:

       no-auto-default
           Specify devices for which NetworkManager shouldn't
           create default wired connection (Auto eth0). By
           default, NetworkManager creates a temporary wired
           connection for any Ethernet device that is managed
           and doesn't have a connection configured. List a
           device in this option to inhibit creating the
           default connection for the device. May have the
           special value * to apply to all devices.
    
           When the default wired connection is deleted or
           saved to a new persistent connection by a plugin,
           the device is added to a list in the file
           /run/NetworkManager/no-auto-default.state to
           prevent creating the default connection for that
           device again.
    
           See the section called “Device List Format” for
           the syntax how to specify a device.
    
           Example:
    
               no-auto-default=00:22:68:5c:5d:c4,00:1e:65:ff:aa:ee
               no-auto-default=eth0,eth1
               no-auto-default=*
    
       ignore-carrier
           This setting is deprecated for the per-device
           setting ignore-carrier which overwrites this
           setting if specified (See ignore-carrier).
           Otherwise, it is a list of matches to specify for
           which device carrier should be ignored. See the
           section called “Device List Format” for the syntax
           how to specify a device. Note that master types
           like bond, bridge, and team ignore carrier by
           default. You can however revert that default using
           the "except:" specifier (or better, use the
           per-device setting instead of the deprecated
           setting).
    

Diagnostic Steps

  • Verify the current configuration of NetworkManager and see that ignore-carrier=* is set:

    [root@r74 ~]# NetworkManager --print-config
    # NetworkManager configuration: /etc/NetworkManager/NetworkManager.conf (lib: 00-server.conf, 10-slaves-    order.conf)
    
    [main]
    # rc-manager=file
    # auth-polkit=true
    # dhcp=dhclient
    no-auto-default=*
    ignore-carrier=*
    slaves-order=index
    plugins=ifcfg-rh
    
    [logging]
    # backend=syslog
    # audit=false
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

1 Comments

This solution turns out to be critical with some HA clusters. Having NetworkManager down the interface when the link status changes really messes up pcs, and disabling NetworkManager for the interface isn't an option when there are systemd-controlled daemons that bind to a specific IP (the daemons will fail at boot unless the network-online.target is used).