How to Enable Trace Logging Verbosity in Network Manager on Openshift

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.10, 4.11, 4.12, 4.14

Issue

  • In order to diagnose and resolve problems related to NetworkManager, enabling trace logging verbosity is of great help to understand what NetworkManager is doing. It is advisable to adjust the log level verbosity according to the specific issue that needs to be investigated.
  • If you need to report a bug, just enable domains=ALL:TRACE to collect all trace information to further investigation.

Resolution

It is possible to reconfigure logging at runtime by using the command nmcli general logging level "TRACE" domains "ALL". However, it is often preferable to obtain a complete log from the beginning, particularly when debugging an issue. It is recommended to activate debug logging in NetworkManager.conf or adding configuration files in /etc/NetworkManager/conf.d directory (modify the default behavior of NetworkManager without modifying the main configuration file directly) and restart the service.

To enable trace logging verbosity level in NetworkManager on Openshift for worker nodes, follow these steps:

  • Create a MachineConfig to modify the log settings to enable trace logging verbosity
$ cat <<EOF > nm-trace-logging.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-nm-trace-logging
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,W2xvZ2dpbmddCmRvbWFpbnM9QUxMOlRSQUNFCg==
        mode: 0644
        overwrite: true
        path: /etc/NetworkManager/conf.d/99-nm-trace-logging.conf
EOF
  • Verify the new NetworkManager configuration file contents /etc/NetworkManager/conf.d/99-nm-trace-logging.conf to be deployed
$ cat nm-trace-logging.yaml | awk -F'base64,' '{print $2}' | base64 -d
[logging]
domains=ALL:TRACE
  • Apply the new MachineConfig 99-nm-trace-logging to enable trace logging verbosity
$ oc apply -f nm-trace-logging.yaml

Once the MachineConfig is created, the Openshift MachineConfig Operator (MCO) will drain/restart the Openshift worker nodes and NetworkManager service will apply the new settings.

$ oc get nodes
NAME       STATUS                     ROLES    AGE    VERSION
master-0   Ready                      master   133m   v1.23.12+8a6bfe4
master-1   Ready                      master   133m   v1.23.12+8a6bfe4
master-2   Ready                      master   133m   v1.23.12+8a6bfe4
worker-0   Ready                      worker   115m   v1.23.12+8a6bfe4
worker-1   Ready                      worker   113m   v1.23.12+8a6bfe4
worker-2   Ready,SchedulingDisabled   worker   115m   v1.23.12+8a6bfe4 << draining/restarting worker
  • Verify the MachineConfig resource 99-nm-trace-logging is created and the MachineConfig is applying changes (UPDATING=True)
$ oc get machineconfig | grep 99-nm-trace-logging
99-nm-trace-logging                          
$ oc get machineconfigpool worker
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED ...   
worker   rendered-worker-72016d37e7b7b1dac230ad0b2e18ee98   True      True      False    ...

In this case, worker nodes will be restarted one by one if the MaxUnavailable is set to 1.

  • Once the changes han been applied and the MachineConfigPool updating status is false, verify that logging verbosity level has been changed on a worker node.
$ oc debug node/worker-2
sh-4.4# chroot /host bash
[root@worker-2 /]# cat /etc/NetworkManager/conf.d/99-nm-trace-logging.conf 
[logging]
domains=ALL:TRACE

$ journalctl -u NetworkManager -b
...
-- Logs begin at Thu 2023-04-06 11:11:57 UTC, end at Thu 2023-04-06 11:53:04 UTC. --
Apr 06 11:50:20 localhost systemd[1]: Starting Network Manager...
Apr 06 11:50:20 localhost NetworkManager[1345]: <debug> [1680781820.1921] CONFIG: 
Apr 06 11:50:20 localhost NetworkManager[1345]: <debug> [1680781820.1921] CONFIG: [logging]
Apr 06 11:50:20 localhost NetworkManager[1345]: <debug> [1680781820.1921] CONFIG:   domains=ALL:TRACE
...

Documentation references:

Diagnostic Steps

  • LEVEL. The available options for the NetworkManager logging verbosity level
Verbosity Description
OFF OFF disables all logging.
INFO INFO is the default verbosity for regular operation.
TRACE TRACE is for debugging.
DEBUG DEBUG is between TRACE and INFO, but it's too verbose for regular operation and lacks possibly interesting messages for debugging.
WARN WARN logs messages indicate potential issues or problems that could lead to errors if not addressed. Messages logged at this level are less severe than ERR level messages.
ERR ERR logs messages indicate critical errors. This level is useful for identifying severe problems such as network connection failures or security issues

  • DOMAINS. Also in the logging section for NetworkManager, domains let you filter different categories of logs that can be enabled or disabled independently. Examples of domains include wifi, dhcp, ppp, vpn, and device. By specifying the desired domains, you can customize the types of logs that are generated by NetworkManager, which can be useful for isolating issues related to specific aspects of network management.

  • You will find the logfiles in syslog, for example:

$ oc debug node/worker-0
sh-4.4# chroot /host bash
[root@worker-0 /]# journalctl -u NetworkManager -b
  • If the log level is set to TRACE, it is important to note that systemd-journald may drop messages. To prevent this from happening, it may be necessary to disable rate-limiting by setting RateLimitBurst to 0 in the /etc/systemd/journald.conf configuration file and restart the systemd-journald service. To see an example of how to configure journald settings in OpenShift 4, you can refer to the following solution How to configure journald settings in OpenShift 4.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments