nmcli changes do not persist between reboots

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.7
    • 4.8
    • 4.9

Issue

When manually making network changes to a Red Hat CoreOS OpenShift node using nmcli the change is lost after rebooting.

Resolution

  • This issue was fixed for some clusters in 4.10.3 with Pull Request 2742

  • If your version is affected, copy the contents of the tmpfs mount to the persistent filesystem using the following command:

nmcli connection modify <con_name> ipv6.method dhcp
cp /etc/NetworkManager/systemConnectionsMerged/<con_name>.nmconnection /etc/NetworkManager/system-connections/<con_name>.nmconnection`

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Root Cause

This Issue was introduced due to changes made to CoreOS and the NetworkManager config for the NMState operator. This included creating an overlay filesystem that allowed nmstate to write network manager configurations without persisting them to disk, this would allow changes to automatically revert on reboot. However, this has the side effect of also affecting nmcli changes.

More details can be found in Bug Report: 1970021.

Diagnostic Steps

To check if a cluster is affected by the overlay filesystem, run the below command. An example of the output is shown.

If there is no output, this issue does not apply to the cluster.

# mount | grep NetworkManager
overlay on /etc/NetworkManager/systemConnectionsMerged type overlay (rw,relatime,seclabel,lowerdir=/etc/NetworkManager/system-connections,upperdir=/run/nm-system-connections,workdir=/run/nm-system-connections-work)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments