Docker service nova_compute stuck at restarting state

Issue

• nova_compute is stuck at restart state. Tried restarting docker service, but it goes to restarting state again. Here is the result of docker ps -a command:

CONTAINER ID        IMAGE                                                                             COMMAND                  CREATED             STATUS                          PORTS               NAMES
71c4df37f0b2        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "/usr/bin/bootstra..."   3 days ago          Exited (0) 3 days ago                               nova_cell_v2_discover_hosts
7c4b3d1b711f        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "/docker-config-sc..."   3 days ago          Exited (0) 3 days ago                               nova_wait_for_compute_service
6ba775333e37        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "kolla_start"            3 days ago          Restarting (0) 11 minutes ago                       nova_compute
1e0f741087bb        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "/docker-config-sc..."   3 days ago          Exited (0) 3 days ago                               nova_wait_for_placement_service
cfeb3913158f        satellite.mgmt:5000/osp13_containers-nova-libvirt:13.0-90.1560798890     "/bin/bash -c '/us..."   3 days ago          Exited (0) 3 days ago                               nova_libvirt_init_secret
c6a28fc55423        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "/docker-config-sc..."   3 days ago          Exited (0) 3 days ago                               nova_statedir_owner
e7a5441f06a2        satellite.mgmt:5000/osp13_containers-neutron-server:13.0-79.1557945105   "puppet apply --mo..."   3 days ago          Exited (0) 3 days ago                               neutron_ovs_bridge
b597231054df        satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438     "/bin/bash -c 'cho..."   3 days ago          Exited (0) 3 days ago                               nova_compute_init_log

• The following error are also seen in /var/log/containers/nova/nova-compute.log:

2020-04-07 15:20:22.443 1 DEBUG nova.virt.libvirt.host [-] Connecting to libvirt: qemu:///system _get_new_connection /usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py:366
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 443, in get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     conn = self._get_connection()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 426, in _get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     {'msg': ex})
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     self.force_reraise()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     six.reraise(self.type_, self.value, self.tb)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 415, in _get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     self._wrapped_conn = self._get_new_connection()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     wrapped_conn = self._connect(self._uri, self._read_only)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 225, in _connect
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     libvirt.openAuth, uri, auth, flags)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     rv = execute(f, *args, **kwargs)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     six.reraise(c, e, tb)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     rv = meth(*args, **kwargs)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 104, in openAuth
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host 
2020-04-07 15:20:22.450 1 ERROR oslo_service.service [req-b51fee21-2b84-453e-afe0-a47c8245ded7 - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-compute-0.localdomain
2020-04-07 15:20:22.450 1 ERROR oslo_service.service Traceback (most recent call last):
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     service.start()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 161, in start
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     self.manager.init_host()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1170, in init_host
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     self.driver.init_host(host=self.host)
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 492, in init_host
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     self._do_quality_warnings()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 472, in _do_quality_warnings
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     caps = self._host.get_capabilities()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 627, in get_capabilities
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 452, in get_connection
2020-04-07 15:20:22.450 1 ERROR oslo_service.service     raise exception.HypervisorUnavailable(host=CONF.host)
2020-04-07 15:20:22.450 1 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-compute-0.localdomain
2020-04-07 15:20:22.450 1 ERROR oslo_service.service 
2020-04-07 15:20:22.451 1 DEBUG oslo_concurrency.lockutils [req-a32b0ea0-277e-443a-8473-66b835c691c8 - - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2020-04-07 15:20:22.451 1 DEBUG oslo_concurrency.lockutils [req-a32b0ea0-277e-443a-8473-66b835c691c8 - - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:228
2020-04-07 15:20:22.452 1 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied'