Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 43. Introduction to NetworkManager Debugging

Increasing the log levels for all or certain domains helps to log more details of the operations that NetworkManager performs. You can use this information to troubleshoot problems. NetworkManager provides different levels and domains to produce logging information. The /etc/NetworkManager/NetworkManager.conf file is the main configuration file for NetworkManager. The logs are stored in the journal.

43.1. Introduction to NetworkManager reapply method

The NetworkManager service uses a profile to manage the connection settings of a device. Desktop Bus (D-Bus) API can create, modify, and delete these connection settings. For any changes in a profile, D-Bus API clones the existing settings to the modified settings of a connection. Despite cloning, changes do not apply to the modified settings. To make it effective, reactivate the existing settings of a connection or use the reapply() method.

The reapply() method has the following features:

  1. Updating modified connection settings without deactivation or restart of a network interface.
  2. Removing pending changes from the modified connection settings. As NetworkManager does not revert the manual changes, you can reconfigure the device and revert external or manual parameters.
  3. Creating different modified connection settings than that of the existing connection settings.

Also, reapply() method supports the following attributes:

  • bridge.ageing-time
  • bridge.forward-delay
  • bridge.group-address
  • bridge.group-forward-mask
  • bridge.hello-time
  • bridge.max-age
  • bridge.multicast-hash-max
  • bridge.multicast-last-member-count
  • bridge.multicast-last-member-interval
  • bridge.multicast-membership-interval
  • bridge.multicast-querier
  • bridge.multicast-querier-interval
  • bridge.multicast-query-interval
  • bridge.multicast-query-response-interval
  • bridge.multicast-query-use-ifaddr
  • bridge.multicast-router
  • bridge.multicast-snooping
  • bridge.multicast-startup-query-count
  • bridge.multicast-startup-query-interval
  • bridge.priority
  • bridge.stp
  • bridge.VLAN-filtering
  • bridge.VLAN-protocol
  • bridge.VLANs
  • 802-3-ethernet.accept-all-mac-addresses
  • 802-3-ethernet.cloned-mac-address
  • IPv4.addresses
  • IPv4.dhcp-client-id
  • IPv4.dhcp-iaid
  • IPv4.dhcp-timeout
  • IPv4.DNS
  • IPv4.DNS-priority
  • IPv4.DNS-search
  • IPv4.gateway
  • IPv4.ignore-auto-DNS
  • IPv4.ignore-auto-routes
  • IPv4.may-fail
  • IPv4.method
  • IPv4.never-default
  • IPv4.route-table
  • IPv4.routes
  • IPv4.routing-rules
  • IPv6.addr-gen-mode
  • IPv6.addresses
  • IPv6.dhcp-duid
  • IPv6.dhcp-iaid
  • IPv6.dhcp-timeout
  • IPv6.DNS
  • IPv6.DNS-priority
  • IPv6.DNS-search
  • IPv6.gateway
  • IPv6.ignore-auto-DNS
  • IPv6.may-fail
  • IPv6.method
  • IPv6.never-default
  • IPv6.ra-timeout
  • IPv6.route-metric
  • IPv6.route-table
  • IPv6.routes
  • IPv6.routing-rules

Additional resources

  • nm-settings-nmcli(5) man page

43.2. Setting the NetworkManager log level

By default, all the log domains are set to record the INFO log level. Disable rate-limiting before collecting debug logs. With rate-limiting, systemd-journald drops messages if there are too many of them in a short time. This can occur when the log level is TRACE.

This procedure disables rate-limiting and enables recording debug logs for the all (ALL) domains.

Procedure

  1. To disable rate-limiting, edit the /etc/systemd/journald.conf file, uncomment the RateLimitBurst parameter in the [Journal] section, and set its value as 0: 

    RateLimitBurst=0

  2. Restart the systemd-journald service. 

    # systemctl restart systemd-journald

  3. Create the /etc/NetworkManager/conf.d/95-nm-debug.conf file with the following content: 

    [logging]
domains=ALL:TRACE

    The domains parameter can contain multiple comma-separated domain:level pairs.

  4. Restart the NetworkManager service. 

    # systemctl restart NetworkManager

Verification

  • Query the systemd journal to display the journal entries of the NetworkManager unit: 

    # journalctl -u NetworkManager
...
Jun 30 15:24:32 server NetworkManager[164187]: <debug> [1656595472.4939] active-connection[0x5565143c80a0]: update activation type from assume to managed
Jun 30 15:24:32 server NetworkManager[164187]: <trace> [1656595472.4939] device[55b33c3bdb72840c] (enp1s0): sys-iface-state: assume -> managed
Jun 30 15:24:32 server NetworkManager[164187]: <trace> [1656595472.4939] l3cfg[4281fdf43e356454,ifindex=3]: commit type register (type "update", source "device", existing a369f23014b9ede3) -> a369f23014b9ede3
Jun 30 15:24:32 server NetworkManager[164187]: <info>  [1656595472.4940] manager: NetworkManager state is now CONNECTED_SITE
...

43.3. Temporarily setting log levels at run time using nmcli

You can change the log level at run time using nmcli. However, Red Hat recommends to enable debugging using configuration files and restart NetworkManager. Updating debugging levels and domains using the .conf file helps to debug boot issues and captures all the logs from the initial state.

Procedure

  1. Optional: Display the current logging settings: 

    # nmcli general logging
  LEVEL  DOMAINS
  INFO   PLATFORM,RFKILL,ETHER,WIFI,BT,MB,DHCP4,DHCP6,PPP,WIFI_SCAN,IP4,IP6,A
UTOIP4,DNS,VPN,SHARING,SUPPLICANT,AGENTS,SETTINGS,SUSPEND,CORE,DEVICE,OLPC,
WIMAX,INFINIBAND,FIREWALL,ADSL,BOND,VLAN,BRIDGE,DBUS_PROPS,TEAM,CONCHECK,DC
B,DISPATCH

  2. To modify the logging level and domains, use the following options:

    • To set the log level for all domains to the same LEVEL, enter: 

      # nmcli general logging level LEVEL domains ALL

    • To change the level for specific domains, enter: 

      # nmcli general logging level LEVEL domains DOMAINS

      Note that updating the logging level using this command disables logging for all the other domains.

    • To change the level of specific domains and preserve the level of all other domains, enter: 

      # nmcli general logging level KEEP domains DOMAIN:LEVEL,DOMAIN:LEVEL

43.4. Viewing NetworkManager logs

You can view the NetworkManager logs for troubleshooting.

Procedure

  • To view the logs, enter: 

    # journalctl -u NetworkManager -b

Additional resources

  • NetworkManager.conf(5) man page
  • journalctl(1) man page

43.5. Debugging levels and domains

You can use the levels and domains parameters to manage the debugging for NetworkManager. The level defines the verbosity level, whereas the domains define the category of the messages to record the logs with given severity (level).

Log levelsDescription

OFF

Does not log any messages about NetworkManager

ERR

Logs only critical errors

WARN

Logs warnings that can reflect the operation

INFO

Logs various informational messages that are useful for tracking state and operations

DEBUG

Enables verbose logging for debugging purposes

TRACE

Enables more verbose logging than the DEBUG level

Note that subsequent levels log all messages from earlier levels. For example, setting the log level to INFO also logs messages contained in the ERR and WARN log level.

Additional resources

  • NetworkManager.conf(5) man page