NetworkManager.service failed to loaded properly: Transport endpoint is not connected

Issue

• Some services are failing to restart with an error message similar to this NetworkManager.service failed to loaded properly: Transport endpoint is not connected:

[heat-admin@overcloud-compute-2 ~]$ sudo systemctl restart NetworkManager.service
Failed to restart NetworkManager.service: Unit NetworkManager.service failed to loaded properly: Transport endpoint is not connected.
See system logs and 'systemctl status NetworkManager.service' for details.

• sudo systemctl status for that service also return an error :

[heat-admin@overcloud-compute-2 ~]$ sudo systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
   Loaded: error (Reason: Unit NetworkManager.service failed to loaded properly: Transport endpoint is not connected.)
   Active: active (running) since Wed 2024-09-11 15:18:17 IST; 8 months 17 days ago
     Docs: man:NetworkManager(8)
 Main PID: 1700 (NetworkManager)
    Tasks: 3 (limit: 362170)
   Memory: 16.9M
   CGroup: /system.slice/NetworkManager.service
           └─1700 /usr/sbin/NetworkManager --no-daemon

May 30 14:38:13 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596093.2029] device (vlan1045): carrier: link connected
May 30 14:38:13 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596093.2226] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.2814] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.3362] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.3556] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.3890] manager: (vlan1046): new Generic device (/org/freedesktop/NetworkMana>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.4226] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.4266] device (vlan1046): carrier: link connected
May 30 14:38:15 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596095.4490] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>
May 30 14:38:17 overcloud-compute-2 NetworkManager[1700]: <info>  [1748596097.5151] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifc>

• We can see other services are affected by this issue as well:

# systemctl show service --all | egrep "Names=|LoadError"
...
Names=rhsm.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit rhsm.service failed to loaded properly: Transport endpoint is not connected."
...
Names=udisks2.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit udisks2.service failed to loaded properly: Transport endpoint is not connected."
...
Names=dbus-org.freedesktop.login1.service systemd-logind.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit systemd-logind.service failed to loaded properly: Transport endpoint is not connected."
...
Names=tuned.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit tuned.service failed to loaded properly: Transport endpoint is not connected."
...
Names=certmonger.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit certmonger.service failed to loaded properly: Transport endpoint is not connected."
Names=NetworkManager.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit NetworkManager.service failed to loaded properly: Transport endpoint is not connected."
...
Names=polkit.service
LoadState=error
LoadError=System.Error.ENOTCONN "Unit polkit.service failed to loaded properly: Transport endpoint is not connected."

• We don't see NetworkManager connected to dbus:

[root@overcloud-compute-2 fd]# busctl
NAME                                              PID PROCESS         USER             CONNECTION    UNIT                      SESSION    DESCRIPTION        
:1.1344489                                     791929 libvirtd        root             :1.1344489    run-r81e2893b1f4447bdb0a… -          -                  
:1.166                                           3997 certmonger      root             :1.166        certmonger.service        -          -                  
:1.167                                           3954 tuned           root             :1.167        tuned.service             -          -                  
:1.169                                           4063 polkitd         polkitd          :1.169        polkit.service            -          -                  
:1.4                                             1659 systemd-logind  root             :1.4          systemd-logind.service    -          -                  
:1.4018559                                     696453 systemd         root             :1.4018559    user@0.service            -          -                  
:1.4192603                                     403288 systemd         heat-admin       :1.4192603    user@1000.service         -          -                  
:1.4192686                                     405675 systemd         tripleo-admin    :1.4192686    user@1001.service         -          -                  
:1.4194529                                          1 systemd         root             :1.4194529    init.scope                -          -                  
:1.6141118                                     249366 sssd_ifp        root             :1.6141118    sssd-ifp.service          -          -                  
:1.6141119                                     249703 busctl          root             :1.6141119    sshd.service              -          -                  
:1.834533                                        1472 sedispatch      root             :1.834533     auditd.service            -          -                  
:1.834727                                      327068 rhsm-service    root             :1.834727     rhsm.service              -          -                  
:1.834755                                      328043 udisksd         root             :1.834755     udisks2.service           -          -                  
com.redhat.RHSM1                               327068 rhsm-service    root             :1.834727     rhsm.service              -          -                  
com.redhat.RHSM1.Facts                              - -               -                (activatable) -                         -         
com.redhat.tuned                                 3954 tuned           root             :1.167        tuned.service             -          -                  
org.fedorahosted.certmonger                      3997 certmonger      root             :1.166        certmonger.service        -          -                  
org.fedoraproject.SetroubleshootFixit               - -               -                (activatable) -                         -         
org.fedoraproject.SetroubleshootPrivileged          - -               -                (activatable) -                         -         
org.fedoraproject.Setroubleshootd                   - -               -                (activatable) -                         -         
org.freedesktop.DBus                                1 systemd         root             -             init.scope                -          -                  
org.freedesktop.PackageKit                          - -               -                (activatable) -                         -         
org.freedesktop.PolicyKit1                       4063 polkitd         polkitd          :1.169        polkit.service            -          -                  
org.freedesktop.UDisks2                        328043 udisksd         root             :1.834755     udisks2.service           -          -                  
org.freedesktop.fwupd                               - -               -                (activatable) -                         -         
org.freedesktop.hostname1                           - -               -                (activatable) -                         -         
org.freedesktop.locale1                             - -               -                (activatable) -                         -         
org.freedesktop.login1                           1659 systemd-logind  root             :1.4          systemd-logind.service    -          -                  
org.freedesktop.nm_dispatcher                       - -               -                (activatable) -                         -         
org.freedesktop.portable1                           - -               -                (activatable) -                         -         
org.freedesktop.resolve1                            - -               -                (activatable) -                         -         
org.freedesktop.sssd.infopipe                  249366 sssd_ifp        root             :1.6141118    sssd-ifp.service          -          -                  
org.freedesktop.systemd1                            1 systemd         root             :1.4194529    init.scope                -          -                  
org.freedesktop.timedate1                           - -               -                (activatable) -                         -