Docker service nova_compute stuck at restarting state
Issue
- nova_compute is stuck at restart state. Tried restarting docker service, but it goes to restarting state again. Here is the result of docker ps -a command:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
71c4df37f0b2 satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "/usr/bin/bootstra..." 3 days ago Exited (0) 3 days ago nova_cell_v2_discover_hosts
7c4b3d1b711f satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "/docker-config-sc..." 3 days ago Exited (0) 3 days ago nova_wait_for_compute_service
6ba775333e37 satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "kolla_start" 3 days ago Restarting (0) 11 minutes ago nova_compute
1e0f741087bb satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "/docker-config-sc..." 3 days ago Exited (0) 3 days ago nova_wait_for_placement_service
cfeb3913158f satellite.mgmt:5000/osp13_containers-nova-libvirt:13.0-90.1560798890 "/bin/bash -c '/us..." 3 days ago Exited (0) 3 days ago nova_libvirt_init_secret
c6a28fc55423 satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "/docker-config-sc..." 3 days ago Exited (0) 3 days ago nova_statedir_owner
e7a5441f06a2 satellite.mgmt:5000/osp13_containers-neutron-server:13.0-79.1557945105 "puppet apply --mo..." 3 days ago Exited (0) 3 days ago neutron_ovs_bridge
b597231054df satellite.mgmt:5000/osp13_containers-nova-compute:13.0-87.1560797438 "/bin/bash -c 'cho..." 3 days ago Exited (0) 3 days ago nova_compute_init_log
- The following error are also seen in
/var/log/containers/nova/nova-compute.log
:
2020-04-07 15:20:22.443 1 DEBUG nova.virt.libvirt.host [-] Connecting to libvirt: qemu:///system _get_new_connection /usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py:366
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host [-] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied: libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host Traceback (most recent call last):
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 443, in get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host conn = self._get_connection()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 426, in _get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host {'msg': ex})
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host self.force_reraise()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host six.reraise(self.type_, self.value, self.tb)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 415, in _get_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host self._wrapped_conn = self._get_new_connection()
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 369, in _get_new_connection
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 225, in _connect
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host six.reraise(c, e, tb)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host rv = meth(*args, **kwargs)
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host File "/usr/lib64/python2.7/site-packages/libvirt.py", line 104, in openAuth
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied
2020-04-07 15:20:22.447 1 ERROR nova.virt.libvirt.host
2020-04-07 15:20:22.450 1 ERROR oslo_service.service [req-b51fee21-2b84-453e-afe0-a47c8245ded7 - - - - -] Error starting thread.: HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-compute-0.localdomain
2020-04-07 15:20:22.450 1 ERROR oslo_service.service Traceback (most recent call last):
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service
2020-04-07 15:20:22.450 1 ERROR oslo_service.service service.start()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 161, in start
2020-04-07 15:20:22.450 1 ERROR oslo_service.service self.manager.init_host()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1170, in init_host
2020-04-07 15:20:22.450 1 ERROR oslo_service.service self.driver.init_host(host=self.host)
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 492, in init_host
2020-04-07 15:20:22.450 1 ERROR oslo_service.service self._do_quality_warnings()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 472, in _do_quality_warnings
2020-04-07 15:20:22.450 1 ERROR oslo_service.service caps = self._host.get_capabilities()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 627, in get_capabilities
2020-04-07 15:20:22.450 1 ERROR oslo_service.service xmlstr = self.get_connection().getCapabilities()
2020-04-07 15:20:22.450 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 452, in get_connection
2020-04-07 15:20:22.450 1 ERROR oslo_service.service raise exception.HypervisorUnavailable(host=CONF.host)
2020-04-07 15:20:22.450 1 ERROR oslo_service.service HypervisorUnavailable: Connection to the hypervisor is broken on host: overcloud-compute-0.localdomain
2020-04-07 15:20:22.450 1 ERROR oslo_service.service
2020-04-07 15:20:22.451 1 DEBUG oslo_concurrency.lockutils [req-a32b0ea0-277e-443a-8473-66b835c691c8 - - - - -] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2020-04-07 15:20:22.451 1 DEBUG oslo_concurrency.lockutils [req-a32b0ea0-277e-443a-8473-66b835c691c8 - - - - -] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:228
2020-04-07 15:20:22.452 1 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied'
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.